We are currently investigating an issue with our Telehouse point of presence.
Users are currently unable to make or receive calls.
Update with more info to follow shortly.
UPDATE: 15/09/21 08:24 - The network is now running in its origional state.
UPDATE: 14/09/21 21:35 - A temporary hardware fix has now been implemented and further maintenance will be ongoing throughout the night. We expect service to be fully restored by 6am.
UPDATE: 14/09/21 21:29 - Our remote and on site teams have identified the issue with connectivity which affected our UK data centre locations, and a hardware fix is currently being implemented.
UPDATE: 14/09/21 18:43 - Our NOC team continue to work alongside our on-site engineering team, investigating a P1 issue with connectivity between our UK data centre locations.
UPDATE: 14/09/21 16:54 - Engineers are still working with the Data Centres to resolve this issue.
UPDATE: 14/09/21 16:02 - Engineers are already on site in both London Data Centres working to resolve this issue.
UPDATE: 14/09/21 15:41 - Investigations are still underway, update to follow shortly
We wanted to provide some more information before we marked this incident as closed. On 21/07/2021 at approximately 09:17 customers may have experienced a brief disruption of our services.
The servers which run our services live in two data centres each on opposite sides of London. Under normal circumstances there is a reasonable amount of network traffic going between the two data centres. At 09:17 this morning the supplier who provides the primary connection between the two data centres suffered an issue which caused traffic to be temporarily dropped.
When traffic began to be dropped, the network was working to automatically converge and begin sending traffic over the secondary link between the two data centres. Unfortunately, because the primary link was not marked as down, but was instead silently dropping traffic, this resulted in an influx of traffic bouncing between the two data centres. This increase in traffic resulted in there being a delay to the time it took for the network to converge.
At approximately 09:19 all network traffic was passing again as expected and when the primary link had recovered traffic began automatically using this path.
We did have reports of a few customers being unable to connect to some applications until around 12:00. We do believe this to be an application issue which was triggered by the events on the network. However, the issues after 09:19 were not being explicitly caused by network traffic being dropped. We are troubleshooting these issues at the moment.
Now we have an understanding of the issues this morning we’ll be reviewing the network to see if there is anything which can be done in the future to stop a similar issue happening again.
We’re always striving to keep the platform as stable as possible and would like to apologize for any inconvenience caused this morning.
UPDATE: July 21 09:26
We have been investigating logs and could see there was a network interruption this morning between 09:17 and 09:19. Service should have been operational from 09:19 again for the majority of users however we were seeing reports of some users not being able to login to some applications until around 09:43. We are continuing to investigate with our vendors and will provide a further update later today. Service should be operating as expected at the moment.
We are currently investigating an incident which occurred at 09:17 this morning. Further updates to follow.
UPDATE 15, 2021 | 20:49 GMT+01:00
We’ve been monitoring the status of UC-One Trio for over 24 hours now and are happy that the problem is resolved. The maintenance performed to increase the specification of some servers looks to have been the resolution to the problem. This incident has made us aware of some additional metrics which we’d like to monitor to ensure we are as proactive as possible going forward. Apologies for any inconvenience caused.
UPDATE 13, 2021 | 11:05 GMT+01:00
The maintenance work which was scheduled yesterday evening to address this issue was completed successfully.
We have not seen any instances of the problem this morning and are continuing to monitor.
We’ll mark this incident as resolved once we’ve monitored the situation for 24 hours.
We are still seeing intermittent issues affecting UC-One Trio. Users may be disconnected from the application and then be unable to log back in temporarily.
We believe the issue is due to resource constraints on a specific set of servers and are scheduling in emergency maintenance this evening to rectify the issue.
We’ll continue to monitor the situation and provide a further update tomorrow morning. Apologies for the inconvenience this may be causing.
Resolved | Apr 08, 2021 | 07:55 GMT+01:00
Investigation has identified an element which we have modified to address the issue and this has resolved the issue.
Unity users with the issue should restart the Unity Client.
Investigating - Apr 02, 2021 ¦ 08:00 GMT
We are currently investigating reports of some users being unable to login to the Unity application. The issue presents itself by the application hanging on “checking Broadworks version” on initial start up.
We are currently investigating issues affecting some third party applications including Unity Desktop and UC-One
RESOLVED - Mar 17 - 11:47 GMT -
We can confirm that in the last 24 hours we have not seen error logs and the system is operating as expected. We are now confident that the changes made on 15/03/2021 have resolved the issue. Apologies for the inconvenience this incident may have caused.
UPDATE - Mar 16 - 10:20 GMT -
Modifications were made yesterday evening during a maintenance window to address the continued error messages we were seeing related to UC-One Trio. We are confident that the modifications made yesterday have now resolved the issue and we haven’t seen any errors since. We are continuing to monitor the situation closely and will provide a status update at 11:45 on 17/03/2021.
UPDATE - Mar 15 - 10:12 GMT -
This incident remains active. Unfortunately there have been additional error logs generated on 14/03/2021 which are related to this issue. Various required changes, relating specifically to server side logging, have been identified which we believe will help in resolving the problem. That work is being planned at the moment and will be scheduled in as maintenance this evening.
UPDATE - Mar 13 - 16:07 GMT -
Additional patching was carried out yesterday evening to address error logs which we believe to be related to this incident. As it stands, UC-One Trio is operating as normal and the error logs in question are no longer being generated. We are continuing to monitor the situation whilst we carry on with our overall investigation.
UPDATE - Mar 12 - 22:30 GMT -
This evening we will be patching our XSP servers on further advice from Cisco, due to the issues experienced with UC-One Trio logins experienced on the 9th and 10th, and 12th of March 2021.
UPDATE - Mar 12 - 11:19 GMT -
Unfortunately we experienced a recurrence of this issue this morning between 10:15 and 10:24. This would suggest the patch which was applied was not successful in resolving the problem. We are continuing our investigation to try and get this resolved once and for all. Apologies for the inconvenience which may have been caused.
UPDATE - Mar 11 - 08:59 GMT -
Emergency maintenance was performed yesterday evening to resolve this issue after a new software patch was identified to be required. Service is currently stable and we’ll continue to monitor the situation. A final update will be provided in due course when we close this incident down. Apologies for the inconvenience caused by this incident.
UPDATE - Mar 10 - 13:51 GMT -
The issue affecting UC-One Trio is no longer present and we are continuing to monitor the situation whilst we work with the third party supplier. A further update will be provided in due course.
UPDATE - Mar 10 - 11:14 GMT -
We believe this issue to only be affecting UC-One Trio. Our investigation continues as we liaise with the supplier.
We are currently investigating inbound call failures on our Multi User VoIP+ platform.
Customers will experience inbound calls routing directly to voicemail. We are investigating this at the highest priority.
Please accept our sincere apologies for the inconvenience this will cause.
UPDATE - 15:10 GMT -
The issue has been identified and a fix will soon be implemented.
Please accept our sincere apologies for the inconvenience this has caused.
Monitoring - A fix has been implemented and we believe the issue to be resolved. Please re-test and contact our support teams if you are still having a problem.
Please accept our sincere apologies for the inconvenience this will cause.
We are currently looking into an issue where phones on the VoIP Talk platform are reporting “No Service” and not connecting to the platform.
UPDATE: - Engineers have identified a DNS issue and working on resolving this as quick as possible.
RESOLVED: - The DNS issue has now been resolved.