Facebook is back online following a massive outage that also affected Instagram, WhatsApp, Messenger, and Oculus
Just as Facebook’s Antigone Davis was on CNBC defending the company against a whistleblower’s allegations and its handling of research data suggesting Instagram is harmful to teens, the company’s entire network of services went offline.
The outage began shortly before noon ET and lasted nearly six hours before being resolved. This is Facebook’s worst outage since a 2019 incident knocked the site offline for more than 24 hours, with the downtime wreaking havoc on small businesses and creators who rely on these services for a living.
On Monday evening, Facebook issued an explanation for the outage, stating that it was caused by a configuration issue. According to the company, no user data was compromised.
At 5:30PM ET, after failing all tests for the majority of the day, a test of ISP DNS servers via DNSchecker.org revealed that the majority of them were successfully finding a route to Facebook.com. We were able to resume normal Facebook and Instagram use a few minutes later; however, it may take some time for the DNS fixes to reach everyone.
ENGINEERS FROM FACEBOOK WERE SENT TO THE COMPANY’S US DATA CENTERS TO TRY AND FIX THE PROBLEM.
Facebook communications executive Andy Stone says on Twitter, “We’re aware that some people are having difficulty accessing our apps and products.” We are working hard to restore normalcy as soon as possible, and we apologise for any inconvenience.” Mike Schroepfer, who will step down as CTO next year, tweeted, “We are experiencing networking issues and teams are working as fast as possible to debug and restore as fast as possible.”
The outage has disrupted nearly all of the internal systems used by Facebook employees to communicate and work. Several employees told The Verge that they resorted to communicating via their work-provided Outlook email accounts, despite the fact that employees cannot receive emails from external addresses. Employees who were logged into work tools like Google Docs and Zoom before the outage can continue to use them, but anyone who needs to log in with their work email was blocked.
According to two people familiar with the situation, Facebook engineers have been dispatched to the company’s US data centres to attempt to resolve the issue. This meant that the outage, which was already the worst in years for Facebook, could be extended even further.
A quick look at Down Detector (or your Twitter feed) reveals that the issues were widespread. While it’s unclear why the platforms were unavailable for so many people, their DNS records indicate that, as with last week’s Slack outage, the problem is apparently DNS (it’s always DNS).
Dane Knecht, senior vice president of Cloudflare, notes that Facebook’s border gateway protocol routes — BGP helps networks choose the best path to deliver internet traffic — were suddenly “withdrawn from the internet.” While some have speculated about hackers or an internal protest over last night’s whistleblower report, there is no evidence that anything malicious is to blame.
Instagram.com displayed a 5xx Server Error message, whereas the Facebook site simply stated that something had gone wrong. The issue also affected Oculus, the company’s virtual reality arm. Users could load previously installed games, and the browser worked, but social features and the installation of new games did not.
Update October 4th, 3:37PM ET: Added additional information about the outage.
Update October 4th, 4:15PM ET: Added statement from Facebook CTO Mike Schroepfer, along with internal Facebook updates.
Update October 4th, 5PM ET: Noted outage is still ongoing, added information about the 2019 outage.
Update October 4th, 5:35PM ET: DNS updates suggest Facebook is closing in on a solution.
Update October 4th, 6:08PM ET: Facebook.com is back online.
Update October 4th, 10:29PM ET: Added information about Facebook’s explanation.