Why measuring performance is important

A number of the articles discussed in other post on this blog list measuring the performance of the organisations web activities as being important. I’ve found two articles that deal specifically with why measuring performance is valuable and how to approach governance for this important activity.

Measuring Your Web Site's Return on Investment: Whoever isn't measuring ROI on their Web site is crazy, because it is measurable (Mary K. Pratt, May 07)
Key premises of the article is measuring ROI on websites can be an effective method of assessing performance as well as improving in decisions making. Some relevant points:

• Use ROI as a way of defining what is important to the company – don’t just measure everything
• For ROI to be a useful business objectives of the site must be understood
• ROI is useful for comparing ‘what if’ situations and considering the impact of decisions that are outside of the web site, i.e. cost on other parts the business.

Web analytics governance: who’s in charge? (Hurol Inan, Feb 08)
This blog discussed why good governance of the web analytics is important. The post also includes some examples of how various organisation are approaching governance of analytics. Interesting points include:

• What is measured needs to match the web site strategies
• A lack of governance for the analytic activities can mean inconstant measurement
• Web analytics governance model should be a subset of the general governance structure, with a steering committee charged with the task of defining expectations, granting power and verifying performance.

Some examples of Web Governance

The site webcontent.gov has a section on web governance. I’ve discussed their definition of governance in an earlier post. Another useful resource is the sub-section of sample governance model. The most interesting of these is the International Trade Administration’s approach to web governance. What is strong about this particular model is it clearly establishes the notion of a web channel that encompasses both web content and web based applications.

Governance - more than just a policy manual

David Schatsky, in his first-rate article, Recipe for Web Site Governance, makes a strong case for were possible radically simplifying governance. The other powerful concept presented by David is governance is more than just a manual of standards and guidelines. He neatly introduces some very useful concept including:

1. Effective governance covers too many areas to be easily assigned to one team or areas
2. Codify expertise and rules into business process
3. Use an evidence based approach to decision making

The role of steering committees

Two articles I found advocated a web committee as an important element of effective governance. The organisation I work for used to have a web committee. Apparently this had been successful prior to being disbanded during a restructure a few years ago.

The first article, A tangled web we weaved: Who rules the site? Public affairs, the CIO or both? (Trudy Walsh, Feb 07) noted the emergence of a ‘representative governance structure’, which uses an ‘enterprise wide web council’ to manage areas such as standards, and coordinating and reorganising content;

The second excellent article, Traits of an Effective Web Steering Committee (Jason Burby, Mar 08), points out that ‘stand alone’ web groups encounter problems because for large companies the web now encompasses too many business functions and activities. A Web Steering Committee can be an effective means of balancing the various needs of the company. The committee should be seen as;

"as your board of directors for the Web channel -- senior people with a
strong overall company strategic focus. Its members should help look at
near-term success while also focusing on what's best long-term”

The article also goes on list the “Traits of a Successful, Healthy Web Steering Committee”. This is interesting because I think it covers some of the vital themes which have to be dealt with by effective governance activities. These include:

• Representation from all key groups
• Senior leadership from represented groups
• Methodology for measuring overall site performance
• Process for prioritizing initiatives based on business impact

What does Web Governance look like?

Interestingly I found more articles describing various approaches web site governance than I found defining what governance is. The variety in approaches helped clarified the concept that there was not going to be a tidy model that I could be easily adapt. Instead I try to identify themes and obvious good practice that could be incorporated into an approach for the organisation where I work. The next series of post covers the more interesting articles that I found why I think these are relevant to the discussion on web governances.

Strategic vrs Self-organising models

I found a couple of articles that reported on research conducted at Department of Public and International Affairs (George Mason University). The researchers suggested that two basic approaches to governance have emerged:

1. The “Strategic view” or centralised approach with tight approval process by one designated department
2. A “Self-organising” or looser approach with individuals and departments publishing “within the agency to post the material they deem fit for consumption”

A couple of interesting points

• The strategic approach was emerging as result over concerns about relevance of published materials to agencies missions. However, the downside is this can result in complex and often slow approval processes.
• While the wide use of a looser approach had been an important part of allowing the freedom to experiment and to learn what users wanted

Weakness with this approach
It is important to note the articles reports on research conducted sometime prior to 2006 (the date of the first article). Given that the research is at least a couple of years old it is probable that the models being examined are predominately orientated towards managing content publishing. I suspect that the researchers were unlikely to have found many organisations tackling the more challenging governance issues around online services delivery or social networking.

From what I can gather one of the reasons government agencies are still slow to take up tools such as forums and wikis are concerns over control. For agencies with a ‘command and control’ approach to the web these types of activities present a significant challenge.

References

• Web site governance hotly contested in many agencies, Joab Jackson, June 06
• A tangled web we weaved: Who rules the site? Public affairs, the CIO or both?, Trudy Walsh, Feb 07

Defining Web Governance?

For my purposes I did some trawling for a definition and found the following:

“Web Governance is the structure of people, positions, authorities, roles, reponsibilities, relationships, and rules involved in managing an agency's website(s). The governance structure defines who can make what decisions, who is accountable for which efforts, and how each of the players must work together to operate a website and web management process effectively.”
From What is Web Governance?

I thought this was reasonable. The concept that governance is made up of a range of control mechanisms is particularly useful for organisations undertaking more complex activities. However, with this definition the difference between the activities of “governance” and “management” is not immediately clear. The potential weakness of this it is possible to have a well managed web site while having poor governance. The broader of “Information technology governance” from Wikipedia (1 Dec 2008 ) does a better job of separating these.

“The primary goals for information technology governance are to (1) assure that the investments in IT generate business value, and (2) mitigate the risks that are associated with IT. This can be done by implementing an organizational structure with well-defined roles for the responsibility of information, business processes, applications, infrastructure, etc.”

This resource points out that IT governance is not just good management practices: “IT governance is about the stewardship of IT resources on behalf of the stakeholders who expect a return from their investment”. This will require a little massaging to bring into it line with my work place (a government agencies).

So for the web governance model that I’m developing I’m proposing the following:

1. Governance is the mechanisms (structures, roles, processes, business applications, etc) that an organisation uses to control the web
2. The purpose of this control is to ensure; the various web activities align with business objectives, resources are efficiently used, risks are mitigated
3. Management is the activities involved in; defining, opperationalising or undertaking the mechanisms of governance

About this blog

As part of my work I have been asked to prepare a paper proposing a web governance model for the organisation. The organisation I work for is a large Australian Commonwealth government agency delivering services to the general public and supporting industry. I’m being a bit cagey here about naming the organisation because I want to avoid “you’re not authorised to say that about us in public”. However, I think it is important that I mention this as it is a big part of the context of what is written here. Also I am not a consultant trying to increase my client network or peddle my wears.

As I was doing some background research I found that there was a fair bit of information out there but it never seemed to fit that well with the context of my particular organisation. The real problem is the existing models do not deal well with the complex issues around delivery of government services online. I also posted a message on the Web Standards Group discussion list to see if anyone was having similar problems, well it turns out to be common. So I thought; I’d do some research, write-up some ideas, and invite feedback to see I could get a better understanding of the issues and to test some of my thoughts.

Implied in the following discussion is there is not a simple model of governance that can be easily applied in all situations. Especially in larger organisations with complex web activities. My thought was to try and create a way of highlighting particular problem areas and to introduce what I think might be the critical elements of governance: monitoring performance, ownership, operational control.

So please comment on what’s here; I want your thoughts!

1. Is performance measured using crude numbers such as hits, visits or ranking?

How the performance of a web site is monitored and ultimately used is an indicator of the strategic view of the web in your organisation. Crude numbers don’t say anything particularly useful about the site and if decisions are being made based on these then the organisation is probably flying blind. Worse still these numbers can not be matched in a meaningful way to strategic business objectives so it is difficult to gauge if the organisation’s web activities are adding value.

Good
‘Web Stats’ are analysed and reports are reviewed in finer detail. This could include examining reports such as; entry/exit pages, most popular pages, 404 errors and bad request, path analyses, search engine terms, referring pages, etc. These are generally used by the operational team to ‘tune’ the site. The stats are also used in some way in the decision making and goal setting for the web activities.

Better
Key stakeholders engaged with the analytics, ask questions about their services and content. Simple numbers are questioned and there is more interested how the web presence is meeting user’s needs. To understand if this is happening more complex indicators begin to be used such as user behaviour, engagement level and satisfaction. These are inturn used in some way to help inform request for improvements and plan new developments.

Best
The organisation uses a detailed, comprehensive and multifaceted approach to performance measurements. This starts with identifying strategic objectives and defining how to measure these. A combination of qualitative and quantitative measures are use, for example; web stats ‘analytics’, market research, focus group testing, user surveys, benchmarking, etc. Critically, these are reported to those involved in making strategic decisions and are used as part of an assessment of the cost and value of the web presence, perhaps using a methodology such as ROI.

2. Are marketing, PR, internal communications and/or HR the ‘owner’ of the web?

Who is seen as the owner has a big impact on setting the agenda for the web within the organisation. It also influences how different parts of the organisation engage in the operational aspects of the web.

The challenge if areas like communications or HR ‘own’ the internet or intranet is they tend to be involved in support functions rather then being at the ‘core’ of running the business and the web often reflects this. Typically, the primary focus will be content publishing with perhaps a few supporting applications, such as staff access to HR functions on the intranet. In this situation elements of the governance could be strong but narrow. For example, there will be strict controls over branding and stringent content approval processes. Other important activities that fall outside of traditional content publishing will be poorly governed, including: coding standards, management of the infrastructure, accessibility, application development, etc.

More importantly, for many organisations the web environment is much more then just content and a few support applications. It is place for activities that are more central to the business. The governance needs to be able guide, control and manage the risks associated with this increasingly important roll of the for the organisation’s web presence. In this scenario a simplistic assigned ‘ownership’ approach to governance is likely to be insufficient.

Good
There are clear roles and responsibilities around the operational activities. For larger web sites this is typically achieved using a dedicated web team.

However, the web is seen as being part of supporting broader business activities; not just owned by one or two business area or solely the responsibility of the web team. There are identified business owners for all of the various content areas and services. Critically, these owners are engaged in maintenance and on ongoing development.

Better
There are clear champions of the internet and intranet as business tool. People in these roles have; the ability to engage and influence others in the development of vision and objectives, the authority to make decisions, the personal networks to resolve problems that lie the outside web teams direct control, and finally there are able to assess and manage risks.

There are also mechanisms that are capable of coordinating and assigning ownership to those involved in the various roles from day-to-day operations to senior strategic roles.

Best
The organisation is able to identify and adjust the governance roles required to match the strategic objectives of the various uses of the web sites. For example if the public facing web site only needs to be basic cooperate information and a ‘contact us’ page then roles are keep simple. However, if more complex activities such as online service delivery are being deployed then governance roles are adjusted to reflex the increased risk and the inter-dependence with other parts of the organisation.

3. Does the organisation have rogue sites or business units running there own sites or applications?

Another common governance challenge is caused by individual business areas working independently of the web team and developing their own web applications or publishing sub-sites. These rogue sites are generally easy to spot as they are characterised by quality issues such: poor non-standards compliant coding, accessibility and useability issues, inconsistent branding and “look & feel”, etc. This is because these rogues sites are often able to by-pass the rules (policies, standards, guidelines) and processes which help to ensure the organisation’s web activities meets recognised ‘good practice’.

Governance is not just about roles and ownership; a key aspect is being able to control how the environment is run. The most obvious issue with different areas doing their own thing is the organisation’s web activities become fragmented and difficult to manage. However, this by-passing may be evidence that the existing web governance mechanisms are seen as irrelevant or more alarmingly as a bottleneck to getting an outcome. For governance to be effective it needs to be both viable and relevant.

Good
There are accepted conventions which guide the day-to-day operations of the internet and intranet. Theses are typically comprised of a combination of: recognised guidelines and standards, processes and procedures, and assigned responsibilities. On the whole these conventions are followed regardless of which business area is involved because they are seen having value such as making the things easier or more reliable. Critically, in situations where it is important that rules are strictly followed it is difficult for these to be by-passed, for example; accessibility, branding, security and disaster recovery.

There is probably a fair degree of reliance on individuals with specialist expertise who know the recognised good practises. This allows a degree of flexibility to adapt to emerging business requirements. However, this approach is unlikely to workable in situations characterised by rapid change, complexity or high risk.

Better
Rather then just relying on expert knowledge and accepted good practices much of the control aspects of governance is built into the organisation’s systems and business processes. This approach can take several forms. For example to enforce rules around publishing and web site content changes a CMS with good workflows and access controls is used. Business processes are used to guide other aspects of the environment, for examples ‘change control’, ‘disaster recovery and continuity planning’.

Finally, expert know-how is codified into standards, guidelines and procedures to define rules for managing parts of the environment not covered by systems or processes. Where possible these are kept simple and orientated to managing practical activities. This is not about having a weighty policy manual that attempts to document every aspect of the web environment as these tend to be an end in themselves rather than a practical resource.

Best
The organisation combines all of the above and builds on this further by using recognised methodologies to direct and guide the development of the web environment. Again, this can take a number of forms. For example; a UCD development process can be used to help keep developments focus on user’s needs along with recognised project management methodologies to help focus on business requirements.