Thoughts About Reporting and Monitoring Products for Office 365

Tony Redmond

by Tony Redmond on 6/24/2016

Share this:

Article Details

Date Revised:

Applies to:
365 Command, 4Ward365, Cogmotive Office 365 Reports, ENow Mailscape 365, monitoring, Office 365, Office 365 admin center, Office 365 Manage Plus, Office 365 monitoring, Office 365 reporting, Office 365 reporting web service, Office 365 reports, Promodag Reports for Exchange, reporting, Service Communications API, Service Health Dashboard

I have written before about the use of third-party products to report on various aspects of Office 365. The need for tenants to be accurately informed about what happens inside Office 365 has not diminished and although Microsoft has recently upped their game in terms of standard reports available to tenants through the new version of Office 365 Admin Center (still in preview - Figure 1), the fact remains that these reports are basic in nature rather than the kind of insightful analysis often required by administrators.

Office 365 Admin Center standard report; Office 365 reporting
Figure 1: A standard Office 365 report available from the Admin Center

It should therefore come as little surprise to know that ISVs have identified reporting as a space they can fill and a wide range of reporting products are available for Office 365. The market divides into two distinct categories: pure-play reporting products and products that handle both monitoring and reporting for Office 365.

Reporting products tend to use the Office 365 reporting web service and underlying data mart as the source of tenant data, supplemented with scripts and other custom code to gather information about what’s going on inside Office 365. Because Office 365 stores tenant reporting data for only 180 days, it’s common practice for reporting products to extract data from the data mart and import it into their other repositories to allow for longer-term storage and easier analysis.

Monitoring products are designed to provide tenants with better and more timely information about the quality and performance of Office 365 than is available through Microsoft’s Service Health Dashboard (SHD). Office 365 is so big that it can take several hours for Microsoft to officially acknowledge that a problem exists with an application (as in this example), simply because it takes time to collate incident reports from multiple tenants, perform troubleshooting, and arrive at a conclusion. Indeed, it’s often said that social media are a more reliable sources of information when problems occur inside Office 365 than is the SHD.

To speed matters up, monitoring products establish a tenant-specific view of Office 365 by gathering and analyzing information about the various services (Exchange, SharePoint, Skype, and so on) using a mixture of probes and synthetic transactions to supplement the service information available to tenants through the Office 365 Service Communications API. Because focus is placed on the health data for that tenant, any issue that affects delivery of services is picked up and identified much more quickly than through the SHD.

It can be a crowded and confusing market to navigate when investigating what’s available to report on Office 365. Apart from making the decision whether to use a pure reporting product or to invest in a more comprehensive approach to reporting and monitoring (where monitoring is often the more critical need), another consideration is whether to select a cloud-only product or one that requires components to run on-premises. Cloud-only products require credentials to extract data on behalf of a tenant (usually a service account that doesn’t need a license). On-premises products require some code to be installed and run on a local workstation to interrogate and fetch data from Office 365.

Office 365 Manage Plus and Promodag Reports for Exchange are examples of products in the on-premises camp while Cogmotive (Figure 2) and 365 Command are examples of cloud-only offerings. In the middle, ENow Mailscape 365 and 4Ward365 cover both bases in order to be able to deal with on-premises and hybrid components in addition to Office 365. Mailscape 365 is an excellent example of a product that provides reports but whose specialty is providing a single dashboard for monitoring both on-premises components and Office 365 services. In this space, lining one product up against another is not always an apples-to-apples comparison.

Cogmotive Office 365 Reports; Office 365 reporting; Office 365 monitoring
Figure 2: Cogmotive Office 365 Reports

Office 365 is a fast-moving target. Evidence of that fact is seen in Microsoft’s report that they had made 450 changes to the service in the year to August 2015. I have not noticed any drop in the rate of change since. Many of the changes are to improve and enhance user functionality, such as the recent upgrade of the document libraries used by Office 365 Groups to make them comparable to those used by SharePoint team sites. However, changes do occur that affect reporting. Two recent examples are the change in the way that the Get-MailboxStatistics cmdlet reports system items stored in user mailbox and the sudden and still unexplained hiatus when Skype for Business data suddenly vanished from the Office 365 reporting data mart (a problem that has now been fixed).

Change can always be expected in large and complex systems. Office 365 is a massive and extraordinarily complex system. In fact, it’s staggering that Office 365 works as well as it does (this Ignite 2015 session illuminates some of what Microsoft does to keep Office 365 running smoothly). It’s change of this nature that cause enormous problems when code is run on-premises because administrators are required to download and apply updates to reporting products to make sure that data continues to be reported accurately for their tenant.

Avoiding the need to continually patch servers is one of the big reasons to move workload to the cloud and the same logic applies for reporting products. It’s much easier for cloud-only products to apply fixes and introduce new functionality. This is one of the primary reasons why I prefer reporting products that run in the cloud, as it means that I don’t have to deploy or maintain any server hardware (or install software on an existing server) just to gather reporting data for my Office 365 tenant.

Flexibility is another reason. New reports or updates to existing reports must be released to match changing circumstances inside Office 365 (and to help vendors stay competitive). As we have learned with OWA, it’s much easier to update a browser-based interface with new functionality than it is to deploy them to clients that run on-premises, as proven by how slowly new features appear in Outlook compared to Outlook Web App.

Harnessing crowd power is a third reason. By this I mean that an on-premises based product running on a server inside a corporate datacenter can only ever reflect what is happening inside a single tenant because that’s the data it sees. Cloud-based reporting products can take advantage of the fact that they acquire and analyze on behalf of multiple tenants. If a problem occurs in the data for a single tenant, it might be a problem specific to that tenant. But if the same issue occurs for two thousand tenants, the problem is far more likely to lie somewhere in the bowels of Office 365 and requires a Microsoft fix. Telling Microsoft about a problem that affects thousands of their customers is more likely to lead to a fast resolution than an issue that might only exist in a single tenant.

Other reasons to prefer cloud-based products include avoiding the need to download reporting data periodically from Microsoft while staying always up-to-date and having immediate access to new features (the same “evergreen software” argument advanced for Office 365) is another advantage. Being able to spin up a trial or proof of concept to validate a reporting product is also beneficial. Given the number of products that are available in this space, you should definitely run some tests before selecting a product to ensure that you get the reports you need in the format you want. Make sure, for instance, that the reporting product covers all the parts of Office 365 that are important to you instead of being strong in just one area, such as SharePoint Online or Exchange Online.

Overall, the only argument I can see as to why it’s best to use code running on-premises to report against Office 365 is that it might make a company’s security team happier at the prospect of having the data held on local storage. This is a pretty tenuous argument because of the lengths that cloud companies go to in order to protect the security and privacy of customer data. After all, if customer data is compromised, it could be the death knell for the company that holds the data.

Reporting Office 365 is reasonably straightforward but the situation is different when you want to monitor the service. In this case, you’re probably interested in knowing how different Office 365 applications perform as perceived by your users. There’s no good way of being able to assess performance and availability from a user perspective without deploying components such as agents, probes, and synthetic transactions that mimic work done by users behind your corporate firewall.

In addition, if you’re one of the many Office 365 tenants that run a hybrid organization, you’ll want to monitor things like hybrid connections, directory synchronization, federation for single sign-on, and so on that are used to create the unified organization from cloud and on-premises components. Unless you can figure out some method to monitor hybrid connectivity components in a secure manner from the cloud, it’s more than likely that you’ll end up running some on-premises code. Apart from being able to measure end-to-end availability and performance from user desktop to Office 365 datacenter, probes running within an on-premises environment are extremely valuable in terms of measuring Office 365 responsiveness. As proven in previous major Office 365 incidents such as the seven-hour Exchange Online outage in North America in June 2014, a drop-off in responsiveness is often an indication that something bad is about to happen.

If you’re in the market for a reporting product for Office 365, consider these points as you review the different products that are available:

  • Do you need reporting, monitoring, or both?
  • Is reporting or monitoring more important to you?
  • What reports do you consider most important?
  • What components do you need a product to monitor?
  • Are you a cloud-only shop or do important parts of your infrastructure still run on-premises?
  • Do you need to monitor the components used for hybrid connectivity?

Knowing the answers to these questions will guide you in your choice. But before you commit to anything, make sure that you test the software in your environment to make sure that a product is capable of doing everything the vendor claims it can. Software has a nasty habit of disappointing…

Follow Tony on Twitter @12Knocksinna.

Want to know more about how to manage Office 365? Find what you need to know in “Office 365 for IT Pros”, the most comprehensive eBook covering all aspects of Office 365. Available in PDF and EPUB formats (suitable for iBooks) or for Amazon Kindle.

Topic: Office 365

Sign in with

Or register

  • Hi there,

    Have you considered a third party analytics solution for your SharePoint portal? A solution like CardioLog Analytics drills down deep to provide much deeper insights than those available in SharePoint or Google analytics.Its available as an On-Prem or SaaS deployment. The solution is specifically built for SharePoint, so its tailored to provide analytics specifically on SharePoints architecture. If youre looking for more information regarding your portal visitors, navigation paths, search behaviors, and content performance check out

    Hope this helps!
  • I have been reading your article and scanned the products you linked to, however I cannot find a tool, cmdlet or whatever that can generate a report on the usage of rooms in Office365. Our organization wants to know what the utilization of the different meeting rooms in our organization are. Do you know of any product that can generate this can of report? I really dont want to write that code myself.