Rackspace hosted Exchange suffered a disastrous outage starting December 2, 2022 and is still ongoing since 12:37 AM December 4th. At first referred to as connection and login issues, the assistance was ultimately upgraded to reveal that they were dealing with a security incident.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the morning hours of December 2, 2022. Initially there was no word from Rackspace about what the problem was, much less an ETA of when it would be dealt with.
Customers on Buy Twitter Verified reported that Rackspace was not reacting to support e-mails.
This has actually been quite the day with #Rackspace. Every hosted exchange client has actually been down for 14 hours or two. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they fell victim to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace consumer independently messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the previous 16 hours.
Uncertain how many companies that is, however it’s substantial.
They’re serving a 554 long hold-up bounce so people emailing in aren’t familiar with the bounce for numerous hours.”
The official Rackspace status page used a running upgrade of the interruption but the initial posts had no info besides there was an outage and it was being examined.
The first official upgrade was on December 2nd at 2:49 AM:
“We are investigating a concern that is impacting our Hosted Exchange environments. More information will be posted as they become available.”
Thirteen minutes later Rackspace started calling it a “connectivity issue.”
“We are examining reports of connectivity issues to our Exchange environments.
Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their e-mail customer(s).”
By 6:36 AM the Rackspace updates described the continuous issue as “connection and login concerns” then later on that afternoon at 1:54 PM Rackspace revealed they were still in the “investigation phase” of the interruption, still attempting to figure out what failed.
And they were still calling it “connectivity and login issues” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
Four hours later Rackspace described the scenario as a “significant failure”and started offering their clients totally free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround up until they understood the problem and could bring the system back online.
The main guidance stated:
“We experienced a significant failure in our Hosted Exchange environment. We proactively shut down the environment to prevent any further problems while we continue work to restore service. As we continue to work through the origin of the concern, we have an alternate service that will re-activate your ability to send out and get e-mails.
At no charge to you, we will be providing you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 up until additional notification.”
Rackspace Hosted Exchange Security Occurrence
It was not till almost 24 hours later on at 1:57 AM on December 3rd that Rackspace formally announced that their hosted Exchange service was struggling with a security event.
The announcement further exposed that the Rackspace professionals had actually powered down and detached the Exchange environment.
“After further analysis, we have determined that this is a security incident.
The known impact is separated to a part of our Hosted Exchange platform. We are taking necessary actions to assess and secure our environments.”
Twelve hours later that afternoon they updated the status page with more info that their security group and outdoors specialists were still dealing with solving the interruption.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has not launched information of the security occasion.
A security event usually involves a vulnerability and there are 2 serious vulnerabilities currently in the wile that were patched in November 2022.
These are the two most present vulnerabilities:
Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
A Server Side Demand Forgery (SSRF) attack enables a hacker to read and alter data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an opponent has the ability to run destructive code on a server.
An advisory released in October 2022 explained the effect of the vulnerabilities:
“A verified remote aggressor can perform SSRF attacks to escalate benefits and perform arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted versus Microsoft Exchange Mail box server, the enemy can potentially get to other resources through lateral motion into Exchange and Active Directory site environments.”
The Rackspace blackout updates have not suggested what the specific problem was, just that it was a security event.
The most existing status update as of December 4th specified that the service is still down and clients are motivated to migrate to the Microsoft 365 service.
Rackspace published the following on December 4, 2022 at 12:37 AM:
“We continue to make development in resolving the incident. The availability of your service and security of your information is of high value.
We have actually devoted comprehensive internal resources and engaged first-rate external proficiency in our efforts to decrease negative impacts to clients.”
It’s possible that the above noted vulnerabilities belong to the security incident affecting the Rackspace Hosted Exchange service.
There has actually been no announcement of whether client info has actually been compromised. This event is still continuous.
Featured image by Best SMM Panel/Orn Rin