Connectivity Issue

Uptime Impact: 2 days, 3 hours, 46 minutes, and 51 seconds
Resolved
Resolved

We've now resolved the incident. Thanks for your patience.

There has been no further DDOS traffic identified since Wednesday, and services have progressively restored since then. All auxiliary services, including webhooks have resumed since Friday.

If you are experiencing any type of service issue, please contact our friendly support team.

Avatar for Mitchell Hudson
Mitchell Hudson
Recovering

Throughout yesterday afternoon and overnight, the Distributed Denial of Service (DDoS) attack subsided after peaking yesterday morning. Traffic levels have eased, and we are seeing services return to normal.

Current Status:

  • Services are returning to normal operational levels.
  • Device registration and call establishment are working normally.
  • Some minor call quality degradation may persist.
  • If you are experiencing issues with handset connection, please restart your router and affected handsets.
  • Our team is continuing to closely monitor traffic levels along with our providers and network partners, should the DDoS attack traffic return.

We appreciate your patience and understanding during this disruption. We remain committed to ensuring the stability and security of our services.

A post incident report is being prepared and you can request a copy by emailing support@upstreamtech.io. The report may take some time to complete due to the complexity of the DDoS attack.

Avatar for Jack Taylor
Jack Taylor
Identified

We sincerely apologise for the significant service disruptions you may be experiencing. We are currently managing a large-scale, distributed denial of service (DDoS) attack on our network.

This attack is causing intermittent connectivity issues, particularly for services connecting from outside our direct network. We understand how frustrating this is and are working tirelessly to restore stable service for everyone.

Our carrier team has immediately engaged mitigation services and reported the incident to the relevant authorities. We are, in collaboration with our providers and network partners, actively implementing network upgrades to better handle the DDoS traffic and improve filtering.

We also apologise for the limited communication. For security reasons during this active event, we have been unable to provide the usual level of updates on our status page. We assure you that our primary focus is on resolving the issue.

While we are making progress, the complex nature of the attack means we cannot provide a definitive timeline for full resolution just yet. We are committed to restoring full service as quickly and safely as possible.

Recommendation: We recommend logging into the My Account portal (or email us at support@upstreamtech.io to do this for you) to set up call diversion to a mobile number as a temporary measure. This can help ensure you remain reachable.

Our partners are documenting the event thoroughly to provide a detailed post-incident report once resolved.

We deeply appreciate your patience and understanding as we navigate this challenging situation. We value your business and are dedicated to resolving this attack and restoring the reliable service you expect from us.

Avatar for Jack Taylor
Jack Taylor
Identified

We are observing a significant increase in Distributed Denial of Service (DDoS) traffic this morning. Our technical team and upstream providers are actively working to mitigate this attack and restore services to full capacity.

Customers may experience phone registration failures, call failures or degraded audio quality.

Recommended Action: Customers impacted can configure diversions within the My Account Portal, or by contacting our support team by email to support@upstreamtech.io

We understand the significant impact this event is having and appreciate your patience as we work to resolve this issue. We are committed to restoring your services as quickly as possible.

Avatar for Mitchell Hudson
Mitchell Hudson
Recovering

We're currently the target of a large and sustained Distributed Denial of Service (DDoS) attack and, while voice services have been restored, it is continuing to impact the availability of our self-management web portals.

We want to sincerely thank you for your continued patience as we actively work to mitigate the ongoing DDoS, and we understand the frustration and inconvenience this is causing.

Our team is working tirelessly to implement mitigation measures and bring our primary systems back online as quickly and securely as possible. To ensure you can continue to manage your services, you will shortly be able to access our website and self-service portals as normal.

We will continue to provide updates as the situation evolves.

Thank you again for your understanding and cooperation during this challenging time.

Avatar for Jack Taylor
Jack Taylor
Investigating

Overnight the DDOS attack has intensified to more than 10x our normal traffic levels, resulting in overwhelmed links to our carriers, causing packet loss to those customers. We are waiting on carriers to implement the appropriate DDOS counter-measures to improve connectivity.

Options for affected customers in the meantime: Email your preferred diversion to support@upstreamtech.io; our team will implement it for you. Ensure you list DID (phone number) and destination number (divert to).

We understand the severe impact this is causing to our customers, and we are working around the clock to resolve it. We sincerely apologise for the impact.

Avatar for Jack Taylor
Jack Taylor
Recovering

Our technical team and carrier partners have mitigated the impact of the ongoing distributed denial of service (DDoS) attack, and all services have been stable since approximately 4:30 pm AEST.

The attack is ongoing, and our carrier partners have committed resources to packet filtering, although further impacts are possible, and we will continue to monitor the situation and provide updates as they become available.

Currently, all services are showing normal utilisation levels, however if you are having difficulty with your VoIP device, you may need to restart your router, local DNS server, VoIP Handset, or a combination of these to restore full connectivity.

If you experience continued connectivity issues, or other unexpected platform problems, please contact our support team for assistance.

Avatar for Jack Taylor
Jack Taylor
Identified

Our team, along with our providers, is continuing to actively mitigate an ongoing distributed denial of service (DDoS) attack. We have largely mitigated most international attack traffic; however, we are now also seeing attack traffic originating domestically. Although traffic levels have improved significantly, some calls may still experience degraded audio quality. We are closely monitoring the situation and continuing to work towards full service restoration. Thank you for your continued patience and understanding.

Avatar for Jack Taylor
Jack Taylor
Identified

We are currently aware of an ongoing Denial of Service (DoS) attack impacting services. Our technical teams are actively engaged in mitigation efforts and are collaborating with our providers to implement a permanent resolution. We are monitoring the situation closely and will provide further updates as necessary. Customers will experience some call quality degradation and registration timeouts.

Avatar for Jack Taylor
Jack Taylor
Investigating

We're aware of an issue affecting connectivity from some providers. Customers on affected providers may experience intermittent registration or call failures. Engineers are investigating the issue, with further updates to follow.

Avatar for Jack Taylor
Jack Taylor
Began at:

Affected components
  • VoIP
    • Outbound Calling
    • Inbound Calling
    • Web Portal