Connectivity Issue

Uptime Impact: 1 hour, 21 minutes, and 29 seconds
Resolved
Resolved

This issue has now been resolved and services working normally again.

This morning we experienced an unexpected power interruption to some core routing elements of our network, which caused significant service disruption.

Customers may have experienced handset disconnections (red lights, or no lights on handsets), as well as call failures during this time. At approx. 9:45 AM AEST connectivity was lost to the majority of services across multiple ISP’s. Services were restored intermittently from approximately 10:45AM. Full-service restoration was achieved by approx. 11:25AM AEST. (These times are approximate and will be confirmed in the PIR)

In the event that your phones are still not working, please reboot your handsets and try again.

If, after a reboot, you’re still unable to connect, please reach out to our team on 1300 366 165 or support@upstreamtech.io

We sincerely apologise for this interruption and thank you for your patience while our team restored services.

Emergency maintenance will be scheduled for tonight, as the team works to further assess the details of the cause of the issue and take remedial action to protect the network against this kind of issue.

To summarise the cause - a power outage at SY1 this morning on one feed caused a number of core routers to fail. While network redundancy was in place to protect against a power feed failure, when power was restored one of our core database servers in SY1 caught fire (literally), which caused subsequent failures and resulted in this and other servers needing to be physically removed from service.

Further details will be made available via a PIR, which will be released in the coming days as soon as the details of the cause and resultant action plan are confirmed. The times and details above are approximate and based on the information available to us now, and are subject to change. If you would like to receive a copy of this PIR, please email support@upstreamtech.io

Avatar for Jack Taylor
Jack Taylor
Recovering

We are working to fully restore service as quickly as possible and services are now partially restored. At this stage, please be aware that it may take longer than usual for your phones to connect to the server, and calls may intermittently fail to connect. If your phone is not registered (shows no account, or has no lights), please restart it and try again. Our team is working as quickly as possible to restore full service and a further update and details will be provided as soon as possible.

Avatar for Jack Taylor
Jack Taylor
Identified

We are aware of a data centre power interruption preventing a number of customers accessing their voice services. We are working on this as a matter of urgency and the current estimated restore time is 25 minutes. We will update you further as soon as possible.

Avatar for Jack Taylor
Jack Taylor
Investigating

We are experiencing problems with our voice service right now. We're so sorry about that. Please standby for further updates, and we'll be in touch as soon as we know more.

Avatar for Jack Taylor
Jack Taylor
Began at:

Affected components
  • VoIP
    • Hosted PBX Features
    • Outbound Calling
    • Inbound Calling
    • SMS Services
    • Faxing Services
    • Web Portal