It confuses me when I hear people talk about page views and profile completions and even number of activities or documents posted and equate those metrics
to the success of their SharePoint solution. Don’t get me wrong, these measures can be very important indicators of something positive. But even if 100
percent of your organization has created a user profile and 100 percent of your employees visit the intranet home page every day, that doesn’t mean that
your solution is successful or even that it’s been successfully adopted. At the end of the day, the real goal for your solution is not adoption – it’s
meaningful business improvement.
The metrics you can easily capture from SharePoint, with or without third-party tools like Webtrends, do not typically measure what matters most. Employees
care about working efficiently. Managers care about operating metrics. Executives care about financial metrics. Most organizations already have measurement
programs in place to track these critical business metrics. But, the system metrics we can capture from SharePoint and third-party tools can
provide us with some really great clues that can help get to the real business value of our SharePoint solutions.
In this article, I’d like to explore a
methodology for using system metrics to provide clues to help identify and improve both the adoption and business impact of
your SharePoint solution. The five key elements to this methodology are:
Understand the business problem you are trying to solve
Identify use cases that have meaningful impact
Determine the metrics that align with each use case
Understand your baseline and establish a target
Measure and monitor: Be prepared to change
1. Understand the business problem you are trying to solve
To get credibility in the organization, you need to ensure that the solutions you build with SharePoint demonstrate a positive impact on business
metrics--and not just in a soft or indirect way. This means that you need a comprehensive understanding of the business problem you are trying to solve. This
is especially true if you are implementing the social computing features of SharePoint. One of the reasons many social solutions fail to gain traction is
due to the fact that they are disconnected from the key challenges that drive the organization--for example, they are not promoted as a solution or an
enabler of a major organizational initiative. So, as a first step, you need to make sure you have a clear (and documented) understanding of the real business
problems you are trying to solve.
If you aren’t addressing an important or valuable business problem with your SharePoint solution, you need to go back to the beginning to find that
problem, as well as a business sponsor who is committed to solving that problem. Be sure that you are tying your SharePoint solution to a key
organizational initiative or goal. If not, you are working on a side show-project—one whose funding is going to be at risk no matter what your
measurement program concludes.
Another reason to have a clear connection to business goals is to help make decisions about which potential SharePoint project to implement. All
organizations have limited time, budget, and resources, and most find that there are more possible projects for the SharePoint team than can possibly be
accomplished. Therefore, it's important to have a framework for differentiating among opportunities that are competing for scarce resources.
2. Identify use cases that have meaningful impact
Once you’ve identified the key business problem that you're trying to solve, the next step is to identify specific use cases where using your SharePoint
solution can have an impact on those problems. The collaboration solutions you build with SharePoint can have a large impact on your business, but it’s
important to be practical when you are choosing use cases – because you are not necessarily going to solve every issue with SharePoint alone. Be practical.
Use case example: Support-call deflection
One association project I worked on had a goal of providing a self-service knowledge portal that would allow member organizations to share best practices
and learn from others facing similar challenges. They wanted to prevent users from always having to depend on engaging with the headquarters team answer simple questions.
With limited resources at headquarters, they wanted to focus on more complex issues for their members.
One impact-based use case they
identified was simple support-call deflection. In other words, the use case proposed that reducing the number of simple support calls directed to the
headquarters support staff would free up the HQ support team to work on more complex requests. This organization was also already maintaining a knowledge base of resources, but at a considerable cost. With the new approach, members would find answers to simple questions on
their own via the knowledge portal.
Another key business goal was to build an online,
accessible, and searchable inventory of both curated and crowd-sourced practices. The measurable use case for this business goal was knowledge resource
ownership, which would improve the quality of resources available while decreasing the cost of building and maintaining the knowledge base.
3. Determine the metrics that align with each use case
Each use case should have at least two to three metrics that allow you to create a tangible target objective. For example, the scenario above identified the following metrics for the support-call deflection use case:
Number of routine support calls
Number of complex support calls
Number of successful searches for online content by members (searches with at least one result)
Number of 0 results searches
The first two metrics were already being captured by the support team. If the solution was going to demonstrate business value, we would want the first
number to decrease. We might, however, expect the second number to increase – and measuring the impact of that outcome might ultimately be tied to an
improvement in member satisfaction, yet another overall business goal.
The second two measures are system metrics that serve as proxies for business value.
If users consistently get relevant results for their queries, then the solution should be delivering value. The system metrics just provide clues about value. The only way to know for sure
is to explore search outcomes directly with searchers to determine the specific business
value of results found.
Tip: A very easy way to do this, by the way, is to include an opportunity for users to provide search results feedback directly on the search results page.
Another wayis to send out periodic search impact surveys that specifically ask users what they were able to find and what they did with
the results (i.e., how they measured the impact of what they were able to find).
For the second use case in this example, to increase the shared ownership of knowledge resources, we identified the following metrics:
Number of assets contributed by the association: This metric was expected to decrease over time, to demonstrate that the membership was actively
engaged in building the repository.
Number of assets contributed by members: This metric was expected to increase steadily over the first year. An increase in member-contributed
assets, especially if the contributions were distributed across many members, would be an indicator of engagement on the part of the membership, as well
as interest in creating sustainable value, an important business goal.
Number of members contributing at least one asset: The goal for this metric was to ensure that at least 50 percent of the membership contributed
something within the first year.
Percent of registered members (users with a profile) who posted or replied to posts in an online conversation: This metric was particularly
important in this example because the solution that SharePoint was replacing, a traditional listserv, was very active but tended to be dominated by
a small number of loud voices. The goal of this metric was to continuously monitor participation to both ensure that the conversations weren’t
dominated by a small group but also to identify potential members who the association might want to pro-actively encourage to engage.
Choose the metrics you want to capture in terms of the use cases that are of highest interest to your stakeholders and their business objectives. The
metrics in my examples may be completely irrelevant in your organization – especially if you don’t share the same business goals. It's also important to
pick a small number of metrics that are both relevant to the business and have a more direct relationship to business outcomes – and can be collected at a
relatively low cost.
4. Understand your baseline and establish a target
One major mistake people often make in documenting their measures of success with SharePoint is that they try to demonstrate business impact without
knowing where they started. This approach is doomed to failure. You need to have a good baseline (starting point) or you will have no ability to measure
progress. If you don’t have a great baseline measure—or it’s too late to get one— capture you initial metrics when you launch your solution.
In addition to a baseline measure, you also need a target! Make sure that you get agreement from your stakeholders regarding what the target should be. In
other words, you want to try to get agreement on the definition of success – if we hit this target, then the system is successful.
I can’t emphasize the
importance of this step enough: Make sure that your stakeholders agree on the definition of success. In fact, I often start stakeholder interviews with
If I ask these questions up front, I have a pretty good clue on what I should be counting!
Here are the some of the targets we used for the support call deflection use case:
5. Measure and monitor: Be prepared to change
Metrics are just a tool. The entire goal of your measurement program is to get better – to make changes to your solution, or even your metrics program, so
that you can make better decisions and have a bigger impact on organizational results. One thing is certain: The complex and dynamic nature of pretty much
all organizations means that the SharePoint solutions you build have to be flexible enough to change in order to continue to provide value. The
only way to understand what changes you need to make is to define and execute your measurement plan.
Use your measurement plan to assess how users are taking advantage of the solution. Let it act as an early warning system to identify both new metrics
and new capabilities that can help achieve your business objectives. In other words, use it to gather clues about what you need to change.
When a metric shows an unexpected result, try to find out why the result occurred. Is it due to a one-time event, or does it indicate a trend? Ask
yourself if the result you are seeing tells you that you may be measuring the wrong thing, or whether something is fundamentally wrong with the way
the solution has been developed or deployed. At the very least, metrics will help you get ideas for how to improve your solution. Collect and prioritize
these new ideas and go back to your original plans and assumptions to see if they need to be changed. Then, build consensus on what needs to be changed,
how to make the change, and when and how to introduce the change to your users.
The following table presents a format that I have used to help gain consensus about prospective measurement plans. You can use this format to both create
and document the plan and to provide a report card of progress once your solution is implemented.
ROI Use Case
Help members to share best practices and learn from others facing similar challenges without always having to depend on engaging with the
headquarters team to get answers to simple questions
Simple support call deflection – reduction in the number of simple support calls to HQ
· Number of routine support calls to the HQ support team
10% reduction within 6 months of launch
· Number of searches of online content by members
· 95% of searches return relevant content after 6 months
Number or “0 results” searches
· Number of “0 results” searches trends lower and reaches less than 1/week within 6 months
Key points to take away
Perhaps the most important thing to take away from this discussion is that the only metrics that count are the ones that provide clues, either directly or
indirectly, that help you determine how your solution is delivering meaningful business impact. It’s not always easy to get this information from system metrics,
which is why you should not be afraid to do periodic user surveys to draw out stories that describe how the solution has delivered business value.
Stories are how humans communicate and qualitative stories supplemented with meaningful system metrics are the best way to communicate the value and impact
of your SharePoint solution.
For more information and additional examples of both quantitative and qualitative approaches to capturing business value metrics, please check out my white
paper, A Practical Framework for SharePoint Metrics, available to download on www.susanhanley.com.