July 20, 9:22am BST

Good morning,

We're currently receiving reports regarding a small number of customers being unable to access Outlook or mail clients using Exchange 2016.

Our initial tests are unable to replicate the faults but we are investigating further to verify if the issue is with our systems.

If you are having these issues please raise a case providing the following information:

- Some example users
- OS version and build
- Outlook version and build
- ISP or mobile network provider

We will update this once we have confirmation either way.

July 20, 11:17am BST

Good morning,

Our current results of this issue appears to be a external routing issue with certain network providers. We're investigating this further and are asking customers with open tickets to perform tests along side advising which ISP/Mobile provider they are with so we can pin point this. 

If you don't have a case with us please raise one and advise which provider your customers are with.

July 20, 12:07pm BST

Good afternoon,

Further investigations are still pointing towards this being an external routing issue with some ISPs, if you could please attempt a network trace from desktop devices for those that are affected this will help us pinpoint exactly which providers are affected and how so.

For Exchange 2016 use the below:

tracert -h 30 autodiscover.giacomcp.com > "tracert-Giacomcp.txt"

For Exchange 2013 use the below:

tracert -h 30 autodiscover.cloudplatform1.com > "tracert-Cloudplatform1.txt"

For Exchange 2010 use the below:

tracert -h 30 autodiscover.messageexchange.com > "tracert-Messageexchange.txt"

July 20, 3:38pm BST

Good afternoon,

Our investigations have confirmed as we've suspected that this is an external routing issue. The main ISPs/Network Carriers we've seen that are impacted are EE and O2. Other minor providers also seem to use these as carriers and are also sometimes affected.

Unfortunately there is little we can do to counter-act this due to the nature of the issue. We recommend where possible to attempt to use OWA, or if it's at all possible to change carriers temporarily (although we appreciate that most will not have this option).

We will continue to monitor the situation and provide updates where possible.

July 21, 11:09am BST

Good morning,

We continue to have reports of customers that are unable to access our Exchange services via Outlook (and in some cases OWA) due to an issue caused by external routing providers. As mentioned previously customers that are not on the EE and O2 networks seem to be okay. We do not know what routing EE and O2 use that causes this issue, however we are looking at all possible options to try mitigate this fault where possible. Unfortunately due to the nature of the fault this is extremely difficult to circumvent, but we are continuing to explore options.

If you have not yet raised a ticket advising you have this issue, please do so as we will use this to communicate potential work arounds as needed. When raising tickets please continue to provide the below information as it will still help with our investigation:

- Some example users

- OS version and build

- Outlook version and build

- ISP or mobile network provider

We also need a trace route ran so we can identify the route your connection is attempt. Please run the below in command prompt.

For Exchange 2016 use the below:

tracert -h 30 autodiscover.giacomcp.com > "tracert-Giacomcp.txt"

For Exchange 2013 use the below:

tracert -h 30 autodiscover.cloudplatform1.com > "tracert-Cloudplatform1.txt"

For Exchange 2010 use the below:

tracert -h 30 autodiscover.messageexchange.com > "tracert-Messageexchange.txt"

Please ensure the results are provided to us in your case.

July 21, 4:31pm BST

Good afternoon,

We've had confirmation that the network transfer provider Lumen (formerly Level 3 Communications) had a big issue around the Peterborough area yesterday that affected many providers. They believe that the issue was fixed around 10.30pm last night, although there was a lot of traffic congestion caused by the outage. However, due to ongoing reports of customers still seeing issues, their networks team are investigating the fault still. 

Due to the ISPs involved in supporting our infrastructure and the connections that are reliant on Lumens connectivity, we're confident this is the issue. This also explains why only some carriers are affected as not all carriers will use this transfer provider. We're in constant comms with our ISPs regarding this issue so we can provide updates as soon as possible. 

Any work arounds relating to host files will not work as it affects all of our DCs, however if any customer that is able to use a VPN to route their services alternatively may work. (We understand that this work around will not be viable solution for many unfortunately.)

We apologise for the inconvenience caused. 

July 22, 3:13pm BST

Good Afternoon,

Unfortunately at this current time, we do not have further information regarding the cause of the issue, or have an ETA on when this will be resolved, however rest assured we are in constant communications with our ISPs to identity the cause, and resolve as soon as possible.

 We apologise again for the inconvenience this is causing and we will advise as soon as we have further information regarding this issue.

July 22, 5:19pm BST

Good Afternoon,

We are continuing to investigate this issue with our ISPs.

In order to investigate this further, please can a ticket be raised (if not done so already) and provide the following information:

- Some example users

- OS version and build

- Outlook version and build

- ISP or mobile network provider

For Exchange 2016 use the below:

tracert -h 30 webmail.giacomcp.com > "tracert-Giacomcp.txt"

For Exchange 2013 use the below:

tracert -h 30 cas.cloudplatform1.com > "tracert-Cloudplatform1.txt"

For Exchange 2010 use the below:

tracert -h 30 cas.messageexchange.com > "tracert-Messageexchange.txt"

July 23, 4:38pm BST

Good Afternoon,

We do not have any further updates from our last updates on 22/07/2022, however we are still continuing to investigate all options.

Please do continue to provide the below details and traceroutes, as they are extremely useful for us at this stage:

- Some example users

- OS version and build

- Outlook version and build

- ISP or mobile network provider

July 25, 5:50pm BST

Good afternoon,

We’re still getting customers contacting us regarding this issue asking us when we will resolve “the problem with our platform”. We feel that it would be beneficial to all involved to provide some further detail and re-iterate some earlier information.

To start off with the issue customers were experiencing has been narrowed down to an issue that started last week with the Network Transit Provider “Lumen” (previously Level 3 communications). The issue will affect some services with certain ISPs or network carriers that use this service from Lumen. In this instance the ISPs that appear to be the most effected will be attempting to communicate to our ISPs via that service. It’s worth noting that not all services or communications from your ISP will be running through that network carrier or be affected by the issue.

From further investigations today and feedback from one of our customers that has been speaking to EE, it seems they have a known issue with some “hosts” not being reachable. However, we do not have any technical details on what the issue is or what they are doing with regards to it.

We’ve been working with one of our ISPs whilst investigating and looking for potential workarounds that we can either implement or suggest to yourselves, which is why we’ve been asking for trace routes. This is to help pinpoint the exact location of the fault in the network.  As this fault currently sits outside of both our infrastructure and that of both of our ISPs this information is only allowing us to identify where the fault is, and will not allow us to resolve this ourselves.

Based on current ticket volumes surrounding this case, we believe this is affecting around 1-2% of our total mailbox userbase. This is down to the fact that most carriers are still working with no issue at all when attempting to communicate with our services. We can only assume these carriers are not using the same Network Transit Provider and as such have no issue communicating with our exchange services.

Again, our sincerest apologies to those impacted, however we hope this will provide some clarity whilst we investigate this new information.

As a final roundup please continue to provide all of the below information when raising/updating tickets as it will still help us with our investigations:

- Some example users

- OS version and build

- Outlook version and build

- ISP or mobile network provider

The below will allow us to see the path being taken to connect to us, and potentially show where the connection fails, with this please provide the public IP address of the affected customer so that we can trace back to them and make sure there isn’t an issue with us reaching them:

July 28, 2:22pm BST

Good afternoon,

We're continuing our precautionary investigations and trying to explore every last avenue to assist the external parties involved.

Please can you continue to provide traceroutes as previously outlined as well as providing PCAP logs.

For those who are unfamiliar with the process, we've included the below from a 3rd party article on collecting these using Wireshark (a popular, free network protocol analyser). We have no affiliation with Wireshark. (https://help.salesforce.com/s/articleView?id=000327816&type=1)

"Capturing your traffic with Wireshark

After starting Wireshark, do the following:

Select Capture | Interfaces

Select the interface on which packets need to be captured. This will usually be the interface where the Packet/s column is constantly changing, which would indicate the presence of live traffic). If you have multiple network interface cards (i.e. LAN card and Wi-Fi adapter) you may need to check with your IT administrator to determine the right interface.

Click the Start button to start the capture.

Recreate the problem. The capture dialog should show the number of packets increasing. Try to avoid running any other internet applications while capturing, closing other browsers, Instant messengers etc.

Once the problem which is to be analyzed has been reproduced, click on Stop. It may take a few seconds for Wireshark to display the packets captured.

Save the packet trace in the default format. Click on the File menu option and select Save As. By default Wireshark will save the packet trace in libpcap format. This is a filename with a.pcap extension."

Once you have these please attach these to your case or a new one, if you don't have one.

August 01, 3:37pm BST

Good afternoon,

An update regarding the current Network issue for some users trying to access Hosted Exchange.

Some positive news to start with, we've had some reports from users advising that using the official Mobile Outlook App from the App Store seems to allowing users to connect. We would like to stress that we do not know if this will work for all users but initial reports appear promising.

Regarding Lumen, the network transfer provider, and the ISP issue we don't have much to report. We're still encouraging people to contact their ISPs to report this issue. As we don't have a contract with Lumen we are unable to do anything to get this resolved. 

We still require PCAP logs and Trace routes, which are listed previously in this status history. 

There have been some reports of the traceroutes not working as some users default location is system 32. To overcome this, before running the commands listed previously, please run the following:

cd %userprofile%

This will set the users profile as the default location, where the files will be exported to.

As the situation progresses, we will update you further.

August 05, 9:17am BST

Good morning,

An update regarding work arounds for this issue.

We've had some positive testing so far with users swapping from 4G to 3G resulting in them being able to access their mail accounts. 

We hope this will help more users to be able to connect. We've tested this on Both EE and O2 carriers and seem to have consistent success on both.

As previously stated please also reach out to your ISP and Carrier networks to report the issue if it persists.

As we learn more we will update you further.

August 11, 2:38pm BST

We have seen a pattern of EE customers reporting that they received a message from EE just before they started having issues accessing our Exchange environment.

In this update it mentioned EE had updated their networking settings and to restart their device.

If you or your customers also received this, can you provide an update on your ticket with us to confirm this, whilst we have only seen this with EE we are asking users of O2 also confirm if this was seen.

August 16, 12:00pm BST

Good afternoon,

We have concluded further investigations of our infrastructure and end-to-end packet captures, the conclusion of both is that the issue falls outside of our platform and thus our control. In addition we have used a 3^rd party to complete an independent investigation to which the conclusion was the same as ours.

The issue appears to be isolated to users of EE and/or O2 and we continue to share our findings including the definitive packet captures with both ISPs for them to investigate further.

In the meantime, please see a reminder of the workarounds we currently know to work below:

·         Where possible, use an alternative ISP

·         Where possible connect to a WiFi connection that is not supported by an EE/O2 internet connection

·         Use a “Global” VPN to reroute the traffic away from the affected ISPs infrastructure

o   Not all customers will have a VPN ready to go, however we have had very positive feedback on the use of free VPNs from the Android/IOS app stores

o   We have also seen that a proxy service has resolved the issue, we do have some success with a proxy from internal testing

·         For some users using the Outlook app instead of the default mail apps on handsets has resolved the issue

o   This however has not been proven in all cases and we have only been able to replicate this success on one device on O2

·         Dropping the use of 4G/5G and using “3G Only”

o   This again however is not proven in all cases and we have not been able to replicate the results internally, some of our partners however have reported success

Long-term we are still working with EE/O2 and we ask that you continue with raising/pushing your cases to EE/O2 for investigation and for a permanent resolution.

Please continue to provide us with traceroutes and PCAP logs as they will still assist with our investigations and push back.

We will continue to provide updates as we learn more. 

August 23, 1:11pm BST

Good afternoon,

Our infrastructure team along with a third-party have discovered a workaround that has now been tested and rolled out to all of our data centres.

Any affected users should now be able to connect to their Exchange Mailbox via both EE and O2.

We are still monitoring this and will continue to do so until we are confident all cases have been accounted for with the workaround.