Avoid Slow Office 365 Performance. Where are the bottlenecks?

Hello, I am Maniwa, Business Development Manager. It has been a while since the last blog was released. Recently we often hear about troubles and concerns regarding the Office 365 rollout. One of our recent customers, a foreign company, recently installed our high-quality internet service circuit with guaranteed bandwidth to be fully prepared for the rollout of Office 365.

What are the performance isues with Office 365, anyway? I know there are a lot of solutions for those issues in the market, but what is the real problem? After you read this article, you will likely understand the nature of the Office 365 performance issue. I’ll also try to avoid advertising, but please forgive me for introducing our own service at the end of this article.

First of all, as a disclaimer INAP Japan is not partner of Microsoft. This article is not a formal opinion of Microsoft at all. But don’t worry, I will offer you a full insight into the topic.

Where are the bottlenecks?
There are plenty of possible bottlenecks which disturb the user experience of Office 365. They might be the specification of your computer, your internal LAN, your network devices, internet connectivity or conflicts within your applications. Let’s take a closer look.

Number of TCP connections
Everything starts here. When you start an Office 365 application, it connects with the Office 365 servers in the cloud, generating many TCP connections. Below are the results of netstat on my laptop before accessing Office 365.


C:\Users\kmaniwa>netstat /n

アクティブな接続

プロトコル  ローカル アドレス          外部アドレス        状態

TCP    127.0.0.1:49187        127.0.0.1:49188        ESTABLISHED
TCP    127.0.0.1:49188        127.0.0.1:49187        ESTABLISHED
TCP    192.168.1.12:49182     xxx.100.101.120:443     TIME_WAIT

C:\Users\kmaniwa>

Below are the results of netstat right after accessing Outlook.


C:\Users\kmaniwa>netstat /n

アクティブな接続

プロトコル ローカル アドレス 外部アドレス 状態
TCP 127.0.0.1:49187 127.0.0.1:49188 ESTABLISHED
TCP 127.0.0.1:49188 127.0.0.1:49187 ESTABLISHED
TCP 127.0.0.1:49189 127.0.0.1:49190 ESTABLISHED
TCP 127.0.0.1:49190 127.0.0.1:49189 ESTABLISHED
TCP 192.168.1.12:49191 23.46.211.169:80 ESTABLISHED
TCP 192.168.1.12:49192 66.163.35.36:443 ESTABLISHED
TCP 192.168.1.12:49197 35.156.48.155:80 TIME_WAIT
TCP 192.168.1.12:49201 34.214.252.85:443 TIME_WAIT
TCP 192.168.1.12:49203 117.18.237.29:80 ESTABLISHED
TCP 192.168.1.12:49205 xxx.100.101.114:443 ESTABLISHED
TCP 192.168.1.12:49220 64.94.124.18:443 ESTABLISHED
TCP 192.168.1.12:49237 52.109.120.4:443 ESTABLISHED
TCP 192.168.1.12:49238 52.109.120.4:443 ESTABLISHED
TCP 192.168.1.12:49239 52.109.124.1:443 ESTABLISHED
TCP 192.168.1.12:49240 52.109.124.1:443 ESTABLISHED
TCP 192.168.1.12:49241 20.190.141.178:443 ESTABLISHED
TCP 192.168.1.12:49247 52.114.128.9:443 ESTABLISHED
TCP 192.168.1.12:49250 13.107.136.9:443 ESTABLISHED
TCP 192.168.1.12:49251 13.107.136.9:443 TIME_WAIT
TCP 192.168.1.12:49252 13.107.136.9:443 TIME_WAIT
TCP 192.168.1.12:49258 23.100.101.114:443 ESTABLISHED
TCP 192.168.1.12:49263 13.107.6.163:443 ESTABLISHED
TCP 192.168.1.12:49270 54.230.173.13:443 ESTABLISHED
TCP 192.168.1.12:49271 54.230.173.13:443 ESTABLISHED
TCP 192.168.1.12:49272 54.230.173.13:443 ESTABLISHED
TCP 192.168.1.12:49273 54.230.173.13:443 ESTABLISHED
TCP 192.168.1.12:49274 54.230.173.13:443 TIME_WAIT
TCP 192.168.1.12:49275 54.230.173.13:443 TIME_WAIT
TCP 192.168.1.12:49276 54.230.173.13:443 TIME_WAIT
TCP 192.168.1.12:49286 23.100.101.114:443 ESTABLISHED
TCP 192.168.1.12:49289 13.107.3.128:443 ESTABLISHED
TCP 192.168.1.12:49301 23.100.101.113:443 ESTABLISHED
TCP 192.168.1.12:49305 52.112.64.153:443 ESTABLISHED
TCP 192.168.1.12:49307 52.112.64.153:443 ESTABLISHED
TCP 192.168.1.12:49308 52.112.64.153:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49199 [2404:6800:4004:800::2004]:80 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49200 [2a01:111:f100:6000::4134:b84b]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49204 [2620:1ec:a92::156]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49207 [2600:140b:5000:482::35c1]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49208 [2600:140b:5000:482::35c1]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49209 [2600:140b:5000:482::35c1]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49217 [2600:140b:5000:482::35c1]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49218 [2600:140b:5000:482::35c1]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49228 [2600:140b:5000:488::38f3]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49234 [2600:140b:5000:484::753]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49235 [2600:140b:5000:484::753]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49242 [2620:1ec:a92::156]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49243 [2620:1ec:c::11]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49248 [2603:1040:601::36c]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49253 [2603:1046:c09:1072::2]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49254 [2600:140b:5000:4a1::b34]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49255 [2600:140b:5000:4a6::753]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49259 [2603:1036:302:834::2]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49261 [2603:1046:c09:100e::2]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49262 [2603:1046:c09:1016::2]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49266 [2603:1046:c09:1072::2]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49267 [2603:1046:c09:1072::2]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49268 [2603:1046:c09:1072::2]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49269 [2603:1046:c09:1072::2]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49278 [2600:140b:5000:4a6::753]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49283 [2603:1046:c09:1072::2]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49284 [2a00:1450:400c:c0b::be]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49285 [2404:6800:4004:80e::200e]:80 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49287 [2600:140b:5000:488::38f3]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49288 [2600:140b:5000:488::38f3]:443 TIME_WAIT
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49290 [2606:2800:147:ff8:129b:22eb:20b:1347]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49292 [2606:2800:147:ff8:129b:22eb:20b:1347]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49294 [2606:2800:147:ff8:129b:22eb:20b:1347]:443 TIME_WAIT
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49295 [2606:2800:147:ff8:129b:22eb:20b:1347]:443 TIME_WAIT
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49296 [2606:2800:147:ff8:129b:22eb:20b:1347]:443 TIME_WAIT
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49297 [2603:1047:0:8::12]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49299 [2603:1047:0:9::12]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49303 [2603:1037:0:3::e]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49304 [2603:1037:0:3::e]:443 ESTABLISHED
TCP [2400:4070:2e92:5500:253f:b994:78ac:a154]:49310 [2603:1047:0:8::12]:443 ESTABLISHED

Imagine what this means, connection wise, for a user who uses Outlook, Word, Excel and PowerPoint simultaneously…

On top of that, UDP name resolutions run through your network before generating those TCP connections.

NAT table
Normally your computer is NAT-ed when accessing the internet. So, the router or firewalls which cover all the computers in your office should maintain a huge size for the NAT table. The maximum number of NAT entries that a network device has can differ, depending on the model. If the NAT table overflows, the last unfortunate user can’t access Office 365, or they will experience bad performance due to the poor performance of the network device. The same is true with proxy servers, which get in the way and relay http(s) traffic.

Internet connectivity
All the traffic goes through the internet without exception. The monthly fee of the internet service is either based on its bandwidth (i.e. 10Gbps, 1Gbps and 100Mbps) or by usage-based billing. Therefore, it is important to estimate maximum bandwidth consumption when choosing an internet service. So, you, as an IT admin, will calculate the possible bandwidth of your company’s internet, and will determin that it is 800Mbps. You may think that 1Gbps would be enough. Hey, best-effort internet service is cheaper than guaranteed one, right? They are both 1Gbps. Definitely I’ll take the cheaper one. You go ahead and contract with the ISP and most likely you won’t get that speed. You’ll get lower speed and your board members won’t be able to connect to the video call and they are going to grill you. Why? Because it is “best-effort”. There’s no guarantee that you will achieve maximum speeds if it is best-effort. In contrast, “guaranteed” internet service does guarantee the maximum wire speeds. The fastest best-effort local loop which INAP Japan provides, records 700Mbps+/1Gbps (measured by iPerf in UDP mode). Anyway, please remember that the type of internet service you choose might be the cause for your poor Office 365 performance.

VPN
If a VPN connection is used between your computer and the Office 365 servers, you will most likely have bad performance. For example, your company has multiple branch offices and they are all connected to the head office over internet VPN connections, and the head office has the internet connection for all the branch offices. If the branch offices access Office 365 via the head office, you will have a bad performance.

VPN encrypts each packet of Office 365, so it is no wonder that it would cause poor performance.

Recommendation
Microsoft offers an official list of recommended measures to achieve optimal Office 365 connectivity and performance.

Office 365 Network Connectivity Principles

In short:
1. Don’t use VPN. Allow Office 365 traffic to use direct internet connections.
2. Bypass proxy servers.
3. Configure firewalls to permit Office 365 traffic without inspection.

But, how can you do that? In other words, how can you differenciate Office 365 traffic from generic internet-bound network traffic? It is easier said than done. I will explain in more detail.

Pain in the neck 1
To differentiate Office 365 traffic from generic internet-bound network traffic, check the URL and IP addresses of Office 365 endpoints. You can download the list of Office 365 endpoints here.

For example,

– outlook.office.com
– outlook.office365.com

These two domain names are associated with the following IP addresses.

13.107.6.152/31
13.107.9.152/31
13.107.18.10/31
13.107.128.0/22
23.103.160.0/20
23.103.224.0/19
…
2620:1ec:d::10/128
2620:1ec:d::11/128
2620:1ec:8f0::/46

In total, 442 URL entries and 327 IPv4 entries (updated September 22, 2018)

If you are a router admin, you might feel a bit nauseous, but where is the problem?

If you want to configure your router to send Office 365 traffic to the default gateway and permit direct internet access, other traffic should be routed to the VPN tunnel, which you should specify the URL or IP addresses mentioned earlier. A policy-based routing setting of Cisco will look like this.


ip access-list extended O365-TRAFFIC
 permit tcp 192.168.1.0 0.0.0.255 13.107.6.152 0.0.0.1 eq 443
 permit tcp 192.168.1.0 0.0.0.255 13.107.9.152 0.0.0.1 eq 443
 permit tcp 192.168.1.0 0.0.0.255 13.107.18.10 0.0.0.1 eq 443
 permit tcp 192.168.1.0 0.0.0.255 13.107.128.0 0.0.3.255 eq 443
 permit tcp 192.168.1.0 0.0.0.255 23.103.160.0 0.0.15.255 eq 443
(Omitted below)

route-map O365-RMAP permit 10
 match ip access-list O365-TRAFFIC
 set ip next-hop <Gateway for Office 365 >

interface <Interface for Office 365 >
 ip policy route-map O365-RMAP

If it is a YAMAHA RTX1210, you can write ip filter rules with URL.


ip filter 100 pass * www.outlook.com, outlook.office365.com, c.bing.net, c.microsoft.com, crl.microsoft.com, (Omitted below)

ip route default gateway <Gateway IP for Office 365> filter 100 gateway <Gateway IP for other traffic>

Now you can see that you must write a huge amount of config.

Pain in the neck 2
Suppose you have managed to finish the configuration of all the URLs or IP addresses of Office 365 endpoints. It might be hard to hear but it is not the end of the road. Since “Endpoints data is updated at the beginning of each month with new IP addresses and URLs published 30 days in advance of being active.”

This means that the URLs and IP addresses change every month. You need to modify the config every month.

Nature of Office 365 solution
Now that the nature of Office 365 solution has been unveiled. You an automatically download and implement the URLs and IP addresses of Office 365 endpoints which change every month. That’s it. Depending on service providers, other solutions will be added to it.

INAP Japan’s Office 365 solution
What about INAP Japan? We do provide traffic steering of Office 365 using our managed router with our partners. You can install our Performance IP along with your existing ISP. Then our managed router steers Office 365 traffic and others.

Let’s see what it does technically. I have built a test environment on my desktop just for this blog. As I said earlier, we, at INAP Japan, provide both guaranteed and best effort circuits. We have two test local loops to play with in our office, one is Colt’s leased line and the other is Nuro’s best effort circuit. Both are directly connected to our customer edge routers in our POP in Kasai data center, so global IPs of INAP are directly assigned to the interfaces (They are masked, sorry). The managed router is NEC IX series borrowed from our generous partner. I have configured it to route Office 365 traffic to Nuro circuit and the other to Colt circuit.

First, I have configured the automatic download for the Office 365 endpoints list. This can be achieved by the “url-offload” function which is embedded in the router for Office 365. I have pointed the download URL to the one provided by NEC support.


Router(config)# show url-offload status
Profile-name : OFFICE365-PROF
  XML-List Information:
    URL: https://offload.nw-meister.jp/v1/o365.xml
    Last Request: 2018/09/12 07:45:46
    Last Update : 2018/09/12 07:45:47
    1 request, 1 success, 0 failure, 0 format error
  DataBase Information:
    Update: 2018083000
    Protocol: http, https
    Total 769 entries, URL 442 entries, IPv4 327 entries

Now let’s traceroute with TCP mode (traceroute -T). The port number is 80 by default. Remember, url-offload works only for http and https. So, if you execute traceroute with ICMP, it doesn’t work. You can see that Office 365 traffic is routed to GW2(B.B.B.1) as expected.

Non-Office 365 traffic


$ traceroute -T www.yahoo.com
traceroute to www.yahoo.com (106.10.250.10), 30 hops max, 60 byte packets

1  192.168.1.1 (192.168.1.1)  0.085 ms  0.200 ms  0.199 ms
2  A.A.A.1 (A.A.A.1)  0.835 ms  0.994 ms  1.110 ms
3  (Omitted below)

Office 365 traffic


$ traceroute -T www.office.com
traceroute to www.office.com (13.107.6.156), 30 hops max, 60 byte packets

1  192.168.1.1 (192.168.1.1)  0.093 ms  0.174 ms  0.175 ms
2  B.B.B.1 (B.B.B.1)  0.884 ms  0.982 ms  1.135 ms
3  (Omitted below)

NAT translations of the interface for Office 365. B.B.B.2 is the NAT-ed IP.


Router(config)#  show ip NAT translation GigaEthernet1.0
Interface: GigaEthernet1.0
NAT Cache - 12 entry, 65523 free, 157 peak, 321 create, 0 overflow

Codes: A - ALG, S - Static, Service
Prot Inside Address:Port   Outside Address:Port  Dest Address:Port     Time
tcp  192.168.1.2:65402    B.B.B.2:65402   52.112.64.153:443     899
tcp  192.168.1.2:65406    B.B.B.2:65406   52.112.64.153:443     895
tcp  192.168.1.2:65407    B.B.B.2:65407   52.98.68.178:443      892
tcp  192.168.1.2:65416    B.B.B.2:65416   23.2.138.15:443       897
tcp  192.168.1.2:65419    B.B.B.2:65419   52.109.120.4:443      900
tcp  192.168.1.2:65421    B.B.B.2:65421   52.109.124.1:443      900
tcp  192.168.1.2:65424    B.B.B.2:65424   13.107.6.156:443      898
tcp  192.168.1.2:65425    B.B.B.2:65425   52.109.124.1:443      900
tcp  192.168.1.2:65427    B.B.B.2:65427   23.37.157.82:443      898
tcp  192.168.1.2:65428    B.B.B.2:65428   23.42.113.126:443     898
tcp  192.168.1.2:65429    B.B.B.2:65429   52.114.128.10:443     900
tcp  192.168.1.2:65431    B.B.B.2:65431   23.100.101.113:443    899
tcp  192.168.1.2:65434    B.B.B.2:65434   52.98.64.2:443        59
tcp  192.168.1.2:65438    B.B.B.2:65438   40.101.144.2:443      900
(Omitted below)

Below are the NAT translations of the interface for non-Office 365 traffic. Looks like DNS queries (port 53) goes through this interface since url-offload only works for http and https. A.A.A.2 is the NATted IP. As for the dest IPs, these are for internal use, so I have masked them. You can see that there are some port 53 entries.


Router(config)#  show ip NAT translation GigaEthernet0.0
Interface: GigaEthernet0.0

NAT Cache - 180 entry, 65355 free, 216 peak, 1724 create, 0 overflow
Codes: A - ALG, S - Static, Service

Prot Inside Address:Port   Outside Address:Port  Dest Address:Port     Time
icmp 192.168.1.2:1        A.A.A.2:1       x.x.x.x:0       59
tcp  192.168.1.2:65403    A.A.A.2:65403   x.x.x.x:443      891
tcp  192.168.1.2:65404    A.A.A.2:65404   x.x.x.x:443     866
tcp  192.168.1.2:65405    A.A.A.2:65405   x.x.x.x:443     899
udp  192.168.1.2:137      A.A.A.2:137     x.x.x.x:137     298
udp  192.168.1.2:49233    A.A.A.2:49233   x.x.x.x:53       9
udp  192.168.1.2:49250    A.A.A.2:49250   x.x.x.x:53       8
udp  192.168.1.2:49355    A.A.A.2:49355   x.x.x.x:53       13
udp  192.168.1.2:49399    A.A.A.2:49399   x.x.x.x:53       4
(Omitted below)

Of course, you can go ahead and replace the existing ISP with INAP Japan, but this would have several disadvantages, such as, the global IP would change to INAP’s and it might be too much of a burden.

You can choose from guaranteed bandwidth internet service(also known as “dedicated Internet access (DIA)”), and shared bandwidth internet service(also known as best-effort internet service). However, We recommend the guaranteed type; please feel free to consult with us if you need advice.

If you are not sure what kind of traffic is running through your internet, we can provide a network visualization service. It is an IP flow monitoring tool, using NetFlow and other equal technologies to show you who use what kind of traffic.

As for the automatic download of the Office 365 endpoint list, writing a script is also a good idea. Actually, one of our customers has done this already. They automated the process of downloading the XML endpoint list from the Microsoft’s web site into their firewall. However, for those who are stressed out when thinking about the testing, debugging and maintenance of the script, you might want to consider using the managed router I have introduced here, as the easier choice. I hope this blog will help you rollout Office 365 successfully in your organization.

Corporate Development Dept.
Business Development Manager
Kazuhiro Maniwa

PacketFabric Column & News

Avoid Slow Office 365 Performance. Where are the bottlenecks?