/pol/ ▶ SHTF Anonymous flag/Slovakia 10/04/21(Mon)21:26:47 No.342120975 I'm a current Major Incident Manager for one of the largest NSPs in the world…that each application server cluster and all of their redundant clusters hosting/support each app (FB, IG, WA) all failing concurrently AND physical access controls locked out…this is either a military takeover or Zuckerberg is literally crashing the plane with no survivors.
These apps are down GLOBALLY. There are DCs all over the world hosting these applications regionally…how do ALL of the DCs for each INDIVIDUAL APP fail at the same time?"
DNS is levels above the actual problem. Fb's own DNS servers are not reachable because of an underlying network problem. Apparently a screwed up change to BGP took down their peerings and they stopped announcing ranges where their DNS servers are. That's why it appears as DNS issue to noobs. And now they may have routers all over the globe they cannot get into, and the local guys manning the DCs who have physical access don't have the knowledge to fix this shit themselves. This stuff at this scale is not about loading older configs. The complexity and scale of Fb network is beyond comprehension and you may have to fix literally tens of thousands of devices. Good luck. Like I wrote, depending on the nature of the problem this will take days if not weeks to restore, although some very limited restoration could be expected in 6-12 hours.
.
The follow-up problem is that their DNS entries have timed out from caches and now the whole world, every mobile phone and computer, are DDoSing their authoritative DNS servers. ISPs are reporting significant increase of DNS traffic, all going to Fb. Their circuits and servers are going to be hammered for days to come.
From 2011: "Facebook trapped in MySQL 'fate worse than death'"
.
https://gigaom.com/2011/07/07/facebook-trapped-in-mysql-fate-worse-than-death/
In June, 2021, they started updating to MySQL 8.
.
If it was only internal routing tables deleted, imagine having to reconnect 2000+ machines. It will never happen.
.
They use MySQL as a key-value-store; if the keys are deleted, just one column in the database, it will never come back.
A problem used to be hosting database that consume more memory available on one machine. EBay solved by deleting history. Google solved by inventing "Big Table. FB has been using duct tape. It was a miracle it worked a decade ago. If it's finally gone, no one who understands the system would be surprised. There is no backup.
.
FB, I'm 2015 or so, was storing passwords in plaintext. They grew fast by taking on technical debt. MZ is basically an actor. If it's gone, we'll it lasted longer than anyone should've ever expected.
ConversionConversion EmoticonEmoticon