Monitoring and Evaluating Web Communications

Heather Creech, International Institute for Sustainable Development

Most organizations want to know whether the knowledge of the organization is reaching its target audience through the Web. But the tools available for monitoring and evaluating Web communications are often not used consistently or appropriately.

There are four reasons for investing time and resources in Web site analysis:

1. to gauge whether and how audiences are finding your knowledge on the Web;
2. to evaluate marketing and outreach activities designed to drive target audiences to your Web site;
3. to improve content and navigation on your Web site; and
4. to gauge system performance and technical requirements.

Background reading

• Jakob Nielsen's Designing Web Usability: The Practice of Simplicity.¹
• Nielsen's e-mail newsletter, the Alertbox, can be subscribed to at http://www.useit.com/alertbox.
• The Web site for Analog, a log file analysis software provider, also provides information on how the Web works and important considerations about analysis of Web site statistics and user assessment (http://www.analog.cx/docs/webworks.html).
• Scott Anderson et al., Tools for Assessing Web site usage:² http://www.iisd.org/pdf/2001/Web_evaluation.pdf. This paper covers in more detail how to analyze Web traffic, conduct user surveys and monitor special marketing techniques such as Web ad banners. Other tools such as counters, cookies and log on forms are also explored.

Basic tools and techniques recommended

1. log file analysis of traffic on the Web server;
2. reviewing whether and how your Web site appears in search engines, Web gateways and other directories;
3. user feedback; and
4. user testing.

1. Log file analysis

a. The limitations of log file analysis

Log file analysis does not provide a clear, reliable picture of the number of users of a Web site, either their geographic distribution or sector representation.

• In general, log file analysis provides an overestimation of the numbers of users, because of the difficulty of screening out search engines and other intelligent agents indexing your Web site.
• It also provides an underestimation of users, due to caching of Web pages on intermediary computer servers. Users may work with the cached content rather than access your site directly.
• Most traffic (over 65 per cent) may now be unidentifiable by your Internet Services Provider due to the increase in fire walls and other security measures. This will prevent you from drawing any reasonable conclusions about hosts accessing your site, either by type (.edu, .com, etc.) or by geographic domain.

Site statistics should be used sparingly for corporate promotional purposes, and always with the caveat that it is difficult to claim levels of traffic, geographic and sectoral reach with any definitive degree of accuracy.

b. The uses of log file analysis

Nevertheless, log file analysis can provide useful benchmarks and comparative measures to improve site structure, management and marketing efforts. You should arrange with your Internet Services Provider to give your the raw log file data which you can then process yourself with a Web traffic analysis software program. Alternatively, your ISP may be willing to analyze the traffic with their own program and give you a report monthly or quarterly.

There are a number of software programs available for analyzing Web traffic. Most of these provide far more information than is necessary or reliable, especially for the small- to medium-sized sites managed by civil society organizations. Also, it is now relatively accepted that "hits" should not be used as a measure of traffic. Any given page could include a dozen or more graphic and text elements, each of which counts as a "hit." This figure significantly skews any representation of traffic as an indication of level of use.

Only a few variables really need to be monitored in log files:

• Page requests (as a very rough equivalent to amount of information screened by the user).
• PDF file downloads (as an indication of interest in specific content, and a more accurate indication of content used).
• Subscriptions to e-mail lists and online conferences (Note: this figure is more likely to be available through your electronic list management software package rather than through your Web traffic analysis; but it is included here because it is also an indication of interest in specific content).
• Key words and search queries entered by users in search engines to access your site (as a reflection of the interests of users that led them to your site).
• Broken links (as an indication of regular maintenance needed).
• Processing time (as an indication of the functionality of the site: are the pages loading fast enough, or are graphics, programming scripts, Flash and other design features becoming a barrier to the user rather than an enhancement to the content)?

Use these variables to assess the following:

• Growth in traffic overall: are the number of page requests, file downloads and subscriptions increasing steadily, or have they tapered off? If traffic is tapering off, you need to review the following:
  • Are you are keeping your Web site current with new information to encourage repeat use and to attract new users?
  • Is there a new Web site that holds similar content, which is competing for your niche on the net?
  • Are users finding your site slow to load or difficult to navigate?
• Identification of peak periods during the year when traffic is heavier (e.g., during the academic / school year).
  • These may be the best periods to launch new products, when you have a larger audience coming to your site.
• Spikes in traffic when a new product is launched: Did marketing efforts lead users to the Web site? Does the traffic to that product remain constant after the launch, or does it drop dramatically after the launch? If traffic drops dramatically:
  • While your initial marketing may have been effective, the product itself may not have met the needs/interests of the target audience.
  • Other Web sites may not have linked to your new product.
  • You may have designed your product (deliberately or accidentally) to have a limited "shelf life": an online conference; a set of briefing papers on a current issue; a handbook or other product that would be of interest only for a short period of time.
• Comparison of traffic on different sections of the Web site: Are some databases or other products on a Web site consistently receiving more use than others?
  • This may simply reflect the level of interest in different sections of your site.
  • It may also signal that some sections are becoming dated, or may require new or better meta-tags, or may have some functional problems (databases not working properly; broken links, etc.)
• Review of key words and search queries entered by users in search engines to access your site: Look at keywords in the log files to identify general issues and concepts that are driving traffic to your site. The keywords provide an indication of how successful search engines are at finding your site.
  • Some keywords may be surprising: users may be looking for information about an individual who has the same name as one of your staff; or the query may indicate an interest in something totally different from what you really offer on your Web site. There is little you can do about this.
  • You may also notice that key words you would expect to be used to find your site are not being used. You may discover that you did not actually use those words in the text, headings and meta-tags of your Web pages; and consequently the search engines are not indexing your site the way you would like. It may be as simple as, for example, tagging your site for the phrase "environment and development" but not "sustainable development."
• Point of entry to your site: are users accessing you more frequently by your home page; are users entering the name of your organization or the title of one of your products into search engines in order to find you? This demonstrates an increase in name recognition for your organization.

2. Review how your Web site appears in search engines, gateways and directories

On a regular basis, enter key words into major search engines to determine whether your site is being retrieved for those terms, and where your site sits in the order of sites retrieved for those terms (are you in the top 10? top 50?) If your site is not being retrieved easily, you may need to review how you have meta-tagged your site; you may need to add more specific terms to your tags; you may also need to review the headings and text on your Web pages: are they clearly written; do they include the words that people would use in search engines to find those pages?

Users also go to intermediate gateways and online directories to locate relevant content on the Web. You should become familiar with those in your field, and request that your site be included. If those directories are highly selective, and your request is turned down, find out what you might need to do to improve the content on your site in order for it to be included.

3. User feedback

User feedback forms add a customer relations element to your external contacts, an approach not commonly stressed in NGO Web communications. A feedback mechanism is an easy way to collect users' opinions about your site. It is becoming standard practice to have a "contact us" button on top-level Web pages. This page can include a feedback form that provides users a means of providing comments, complaints and requests for further information on your site. Be sure to acknowledge comments received; and request their permission to add them to your mailing lists in order to keep them interested and engaged in your work.

These issues are discussed in greater depth in Managing user feedback and communications.

4. User testing

User testing is the most effective means to evaluate a user's experience with your site, and will also identify any problems with site functionality. This information should be used to ensure that your site will have the attributes that attract repeat users (i.e., ease of use and quality content). Although user testing might seem daunting, it can take many forms and be as formal or informal as you wish. Nielsen³ notes that useful data can be collected in as little as an hour, and that most major usability problems can be found by testing as few as five users.^4,5

While it is more effective to test representative users from your target audience, don't let this prevent you from conducting user testing. Testing your site with a few staff from within your organization will still provide useful results.

The following simple methodology is based on the "Discount Usability Engineering" model described by Jakob Nielsen.^6,7

• Ask a group of test users to browse the site for five minutes.
• Next, assign general and specific task-related questions based on site content that require different parts of the site to be used (e.g., questions requiring the use of an online calendar of events; or the use of the search functions and navigation to locate specific information).
• Ask each user to "think out loud," verbally noting aspects about the site that they like, don't like, or find confusing or clear as they browse and perform the tasks.
• Record their comments and your own observations while your test group answers the questions, in order to compare comments with actual results of tasks.
• Finally, have users fill out a questionnaire about their impressions of the site (e.g., if the meaning of icons was clear, if navigation was easy, if there was graphic consistency and logical structure, aspects they liked the best/least).

1 Jakob Nielsen, Designing Web Usability: The Practice of Simplicity, New Riders Publishing, Indianapolis, 2000, ISBN 1-56205-810-X.
2 Scott Anderson, Terri Willard, Heather Creech and Deborah Bakker, Tools for Assessing Web Site Usage. IISD, Winnipeg, 2001.
3 Jakob Nielsen, "Survey your Users," Users First, February 18, 1999, (http://www.zdnet.com/devhead/stories/articles/0,4413,2211547,00.html).
4 Jakob Nielsen, "Cost of User Testing a Web Site," Alertbox, May 3, 1998, (http://www.useit.com/alertbox/980503.html).
5 Jakob Nielsen, "Why You Only Need to Test with 5 Users," Alertbox, September 1, 1997, (http://www.useit.com/alertbox/9709a.html).
6 Jakob Nielsen, "Guerrilla HCI: Using discount usability engineering to penetrate the intimidation barrier," 1994, (http://www.useit.com/papers/guerrilla_hci.html).
7 Jakob Nielsen, "Cheap Usability Tests," Users First, March 11, 1999, (http://www.zdnet.com/devhead/stories/articles/0,4413,2224316,00.html).