Roblox Is Back Online: CEO Blames Several Factors and Apologizes for the Service’s Delay
This weekend, Roblox was subjected to the worst kind of Halloween trick, but kids everywhere are in for a treat: Roblox is back online. After being out for more than two days, the game-making site was restored on Sunday afternoon.
Roblox founder/CEO David Baszucki apologized for the lengthy delay in restarting the service in a blog post, blaming the outage on a number of issues.
On Thursday, Roblox users began to experience issues; according to Baszucki, a critical system in the company’s infrastructure became overwhelmed as a result of a tiny bug in our backend service communications when under tremendous pressure. He made it clear that the issue was “not caused by any spike in external traffic or any specific experience.” The issue was instead caused by the increased number of servers in our datacenters. As a result, most Roblox services were unable to communicate and deploy properly.”
Roblox made that difference after users speculated that the suspension was caused by an overabundance of users participating in a Chipotle promotion. Shortly before the blackout, the restaurant chain had started a Halloween event on Roblox that included a giveaway of $1 million worth of free burritos.
“Recovery took longer than any of us would have desired due to the difficulty in diagnosing the actual bug,” Baszucki stated. “We were able to remedy the issue after effectively identifying the root cause through performance optimization, re-configuration, and load scaling back.”
As the firm completes its research of what happened, Baszucki said Roblox will provide a post-mortem with further specifics. For the time being, the company believes no users have lost data. He wrote, “Your Roblox experience should now be completely normal.”
Roblox has over 200 million monthly active users across a variety of platforms, including iOS, Android, tablets, PCs, Xbox, Oculus Rift, and HTC Vive. The game is played for an average of 156 minutes per day by daily active users.
Here is Baszucki’s full blog post:
As most of the Roblox community is aware, we recently experienced an extended outage across our platform. We are sorry for the length of time it took us to restore service. A key value at Roblox is “Respect the Community”, and in this case, we apologize for the inconvenience to our community.
On Thursday afternoon October 28th, users began having trouble connecting with our platform. This immediately became our highest priority. Teams began working around the clock to identify the source of the problem and get things back to normal.
This was an especially difficult outage in that it involved a combination of several factors. A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service communications while under heavy load. This was not due to any peak in external traffic or any particular experience. Rather the failure was caused by the growth in the number of servers in our datacenters. The result was that most services at Roblox were unable to effectively communicate and deploy.
Due to the difficulty in diagnosing the actual bug, recovery took longer than any of us would have liked. Upon successfully identifying this root cause, we were able to resolve the issue through performance tuning, re-configuration, and scaling back of some load. We were able to fully restore service as of this afternoon.
We will publish a post-mortem with more details once we’ve completed our analysis, along with the actions we’ll be taking to avoid such issues in the future. In addition, we will implement a policy to make our creator community economically whole as a result of this outage. There are more details on this to come. As part of our “Respect the Community” value, we will continue to be transparent in our post-mortem.
To the best of our knowledge, there has been no loss of player persistence data, and your Roblox experience should now be fully back to normal. You can always contact our support team if you experience any hiccups using Roblox now or in the future.
We are grateful for the patience and support of our players, developers, and partners during this time.