Update: 15/09/2020 09:30 BST
See below for more detail from Microsoft Azure. BusinessOptix continue to monitor and identify where mitigations for this scenario are feasible.
From Microsoft Azure:
Connectivity Issues - UK South - Mitigated (Tracking ID CSDC-3Z8)
Summary of Impact: Between 13:54 UTC on 14 Sep 2020 and 00:41 UTC on 15 Sep 2020, a subset of customers in UK South may have encountered issues connecting to Azure services hosted in this region. Customers using multiple Availability Zones, or Zone Redundant services may have experienced minimal impact.
Preliminary Root Cause: A cooling loss event occurred when multiple chilled water pumps shut down. This resulted in cooling loss and the internal temperatures for some parts in a single data center began to rise above the operational thresholds in UK South. Automation began shutting down the network, compute, and storage resources to protect data durability.
Mitigation: We undertook various workstreams to bring back connectivity. The site engineers placed the cooling system into manual mode and began to reset the affected pumps to recover the cooling plant. This helped to bring temperatures to safe operational ranges in all the impacted areas of the datacenter by 16:40 UTC.
Once temperatures were within safe thresholds, engineers started to restore power to the affected infrastructure and began a phased approach to bringing this infrastructure back online. Once storage and the networking infrastructure was fully restored, dependent compute scale units began to recover. As compute scale units became healthy, virtual machines and other dependent Azure services recovered.
Next Steps: We sincerely apologize for the impact to affected customers. We will continue to investigate to establish the full root cause and prevent future occurrences. Stay informed about Azure service issues by creating custom service health alerts: https://aka.ms/ash-videos for video tutorials and https://aka.ms/ash-alerts for how-to documentation.
Update: 22:25 BST
The situation is still being monitored but services are currently being restored and impacted libraries are returning to full functionality. We will continue to provide updates.
Notice:
Microsoft Azure (UK South) are currently experienceing service issues. This is affecting several of our customers libraries on the UK.BusinessOptix.com platform. We are monitoring and will post an update in due course as more information comes to light. See the below link for more information from Microsoft Azure:
Comments
0 comments
Article is closed for comments.