Rackspace hosted Exchange suffered a devastating interruption beginning December 2, 2022 and is still ongoing since 12:37 AM December fourth. Initially described as connection and login problems, the assistance was ultimately upgraded to announce that they were dealing with a security occurrence.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the early morning hours of December 2, 2022. At first there was no word from Rackspace about what the issue was, much less an ETA of when it would be resolved.
Clients on Buy Twitter Verified reported that Rackspace was not reacting to support e-mails.
This has actually been rather the day with #Rackspace. Every hosted exchange client has been down for 14 hours or so. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they fell victim to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace customer privately messaged me over social media on Friday to relate their experience:
“All hosted Exchange clients down over the past 16 hours.
Unsure the number of companies that is, however it’s considerable.
They’re serving a 554 long hold-up bounce so people emailing in aren’t knowledgeable about the bounce for several hours.”
The main Rackspace status page offered a running update of the outage but the initial posts had no information aside from there was an outage and it was being investigated.
The very first authorities update was on December second at 2:49 AM:
“We are investigating a problem that is impacting our Hosted Exchange environments. More information will be posted as they become available.”
Thirteen minutes later Rackspace started calling it a “connection concern.”
“We are investigating reports of connectivity concerns to our Exchange environments.
Users might experience an error upon accessing the Outlook Web App (Webmail) and syncing their email client(s).”
By 6:36 AM the Rackspace updates described the continuous issue as “connectivity and login concerns” then later on that afternoon at 1:54 PM Rackspace announced they were still in the “examination phase” of the interruption, still trying to determine what went wrong.
And they were still calling it “connection and login issues” in their Cloud Office environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
Four hours later Rackspace referred to the circumstance as a “substantial failure”and began using their clients free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they understood the issue and might bring the system back online.
The main assistance stated:
“We experienced a substantial failure in our Hosted Exchange environment. We proactively shut down the environment to prevent any more issues while we continue work to restore service. As we continue to work through the origin of the problem, we have an alternate service that will re-activate your ability to send and get e-mails.
At no charge to you, we will be offering you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 until additional notification.”
Rackspace Hosted Exchange Security Incident
It was not up until almost 24 hr later at 1:57 AM on December 3rd that Rackspace formally announced that their hosted Exchange service was suffering from a security event.
The statement even more revealed that the Rackspace service technicians had actually powered down and detached the Exchange environment.
“After more analysis, we have identified that this is a security occurrence.
The recognized effect is isolated to a part of our Hosted Exchange platform. We are taking necessary actions to examine and safeguard our environments.”
Twelve hours later on that afternoon they updated the status page with more details that their security team and outdoors specialists were still working on resolving the outage.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has actually not released details of the security occasion.
A security event normally includes a vulnerability and there are two extreme vulnerabilities presently in the wile that were covered in November 2022.
These are the two most existing vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack permits a hacker to read and change data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an assaulter is able to run harmful code on a server.
An advisory published in October 2022 described the effect of the vulnerabilities:
“An authenticated remote assailant can carry out SSRF attacks to intensify opportunities and carry out arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mailbox server, the opponent can potentially get to other resources via lateral movement into Exchange and Active Directory site environments.”
The Rackspace failure updates have not shown what the specific issue was, only that it was a security event.
The most existing status update since December fourth specified that the service is still down and clients are encouraged to migrate to the Microsoft 365 service.
Rackspace published the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in resolving the event. The availability of your service and security of your data is of high importance.
We have committed substantial internal resources and engaged first-rate external know-how in our efforts to reduce unfavorable impacts to customers.”
It’s possible that the above noted vulnerabilities belong to the security event impacting the Rackspace Hosted Exchange service.
There has been no statement of whether customer information has been compromised. This event is still ongoing.
Included image by Best SMM Panel/Orn Rin