To dispel rumors of a cyberattack or BGP hijack incident that may have caused the recent outage of the Resolver 1.1.1.1, Cloudflare explains in an analysis of the incident that the cause was an internal misconfiguration.
See also: Cloudflare: DDoS attacks reach record rate of 7.3 Tbps
The outage occurred on July 14 and affected most of the service's users worldwide, rendering many online services unavailable. The announcement comes after reports on social media suggested the problem was caused by a BGP hijack attack. Cloudflare's public DNS service 1.1.1.1 was launched in 2018, promising privacy and blazing-fast connectivity for users around the world.
The company explains that behind the outage was a configuration change made on June 6, in preparation for the future Data Localization Suite (DLS), which accidentally linked IP prefixes to an inactive, non-production version of DLS.
On July 14 at 21:48 UTC, a new update added a test location to the inactive DLS service, which caused a global refresh of network settings and enabled the incorrect configuration.
See also: Hackers use fake Cloudflare verification screen
This resulted in the withdrawal of Resolver 1.1.1.1 prefixes from Cloudflare's production data centers and their routing to a single, out-of-service location, rendering the service globally inaccessible. Less than four minutes later, DNS traffic to Resolver 1.1.1.1 began to decline. By 22:01 UTC, Cloudflare had detected the incident and publicly disclosed it.
The misconfiguration was undone at 22:20 UTC and Cloudflare began re-advertising the deprecated BGP prefixes. Full service restoration to all locations was achieved at 22:54 UTC.
The incident affected multiple IP ranges, including 1.1.1.1 (primary public DNS resolver), 1.0.0.1 (secondary), 2606:4700:4700::1111 and 2606:4700:4700::1001 (primary and secondary DNS resolvers over IPv6), as well as other addresses that support routing within infrastructure . In terms of protocol impact, there was a significant reduction in UDP, TCP and DNS-over-TLS (DoT) traffic to the above addresses. However, DNS-over-HTTPS (DoH) remained largely unaffected, as it uses different routing through the cloudflare-dns.com domain.
See also: Cloudflare: Blocks AI data crawlers by default
The incident serves as a valuable lesson for technology companies: even the most carefully designed infrastructures must have strong safeguards, pre-implementation testing, and immediate recovery mechanisms.
Source: bleepingcomputer
