Usability Testing of the Web Content Accessibility Guidelines - Draft Proposal

Last Modified 24 July 2001


David Sloan
Digital Media Access Group, Department of Applied Computing, University of Dundee, Dundee, DD1 4HN, United Kingdom
Telephone +44 1382 345598, fax +44 1382 345509 email:
Professor Helen Petrie
Director, Sensory Disabilities Research Unit and National Centre for Tactile Diagrams, Psychology Department, University of Hertfordshire, Hatfield AL10 9AB, United Kingdom
Telephone +44 1707 284629, fax +44 1707 285059 email:


This project involves evaluating the usability of the Web Content Accessibility Guidelines (WCAG), with a number of distinct user groups for which the Guidelines are intended. The usability evaluation will involve Version 2.0, the proposed revision of the Guidelines. Version 1.0, the current version of the guidelines, will also be used as a comparison.

Significant user groups will be identified and used throughout the testing process, including both novice and experienced web designers.

The testing will concentrate on eliciting qualitative data, although limited quantitative data will also be elicited. The work will follow on from the work carried out on evaluation of Version 1.0 of the WCAG by Colwell and Petrie (1999).

Deliverables are anticipated to include:

  1. Report on knowledge of, and attitudes towards, Web accessibility
  2. Report on the usability evaluation process, plus recommendations for, where appropriate, improving the usability of the WCAG.

It is hoped that results from the study will produce valuable input into the final content and appearance of Version 2.0 of the WCAG.


The main question to be answered is "Are the WCAG usable to the intended audience", bearing mind that the intended audience could be virtually anyone. To answer this question, qualitative and quantitative data, from usability testing with a variety of user groups, will be used to assess current strengths and weaknesses of the two versions of the guidelines.

The findings of the testing stage will be passed back to the Web Accessibility Initiative as part of the development stage of the WCAG version 2.0.

Aims and Methodology of Research

Digital Media Access Group

The Digital Media Access Group (DMAG) is a research group based within the Department of Applied Computing at the University of Dundee. The main focus of the group is accessibility and usability of digital resources, primarily web based resources.

Research has included the development of a methodology for auditing the accessibility of web sites, and for generation of clear and prioritised recommendations aimed at maximising the accessibility of the resource. Members of DMAG have presented at:

DMAG have also supplied accessibility support, in the shape of advice or resource audits, to clients such as:

DMAG Home Page

Sensory Disabilities Research Unit

The Sensory Disabilities Research Unit (SDRU) at the University of Hertfordshire conducts research on how new technologies can assist in overcoming the problems faced by disabled and elderly people in their day to day lives and on how to design new technologies so that they are usable by the broadest possible range of users. Its work specializes in the elicitation of user needs and requirements, the design of new technologies, the adaptation of existing technologies and the evaluation of systems. The SDRU has a multi-disciplinary team of researchers with backgrounds in psychology, computer science and information science, and considerable expertise in research in this area. A high level of end-user involvement takes place in all work and the SDRU has an enthusiastic and highly critical panel of disabled and elderly people who participate in studies. Good links have also been established with many local and national organizations of and for disabled and elderly people.

The SDRU has undertaken a number of activities related to Web accessibility, including:

SDRU Home Page


The profile of web accessibility has never been higher, with the implications of legal requirements following legislation such as Section 508 of the Rehabilitation Act in the US, and the Disability Discrimination Act in the UK, on those involved in web design. The W3C's Web Accessibility Initiative has played an invaluable role in raising awareness in accessible design issues, notably through development of the WCAG.

Yet, it is not yet clear how widely understood the important issue of Web accessibility is, in the UK and elsewhere, amongst the people developing Web resources - either the organizations which wish to have Web sites or those who are able to develop the Web sites. Nor is it clear whether Web developers find the Guidelines understandable and easy to use. An initial investigation of the usability of a draft version of the Guidelines by Colwell and Petrie (1999) 1, albeit undertaking with students who were still learning Web development, showed that there were a number of areas of confusion concerning the Guidelines. This initial conclusion has been reinforced by numerous requests from professional Web developers for clarification of the Guidelines. Furthermore, previous research on the use of guidelines as a tool for computer software and interface developers (for example De Souza and Bevan, 1990 2; Mosier and Smith, 1986 3) has shown that developers had significant difficulties in interpreting and following guidelines.

There is thus a need to ensure that the forthcoming version of the WCAG is designed in such a way as to maximise usability amongst the target audience.

Aims of the Research

The proposed research will undertake two studies, one on the level of knowledge of the Web accessibility, and the other of the usability of the draft revision of the WCAG guidelines with the following aims:

Study 1: Knowledge of and attitudes towards Web accessibility issues amongst the Web authoring community

This study will undertake a survey of the current levels of knowledge of and attitudes towards Web acccessibility issues in organizations wishing to have Web sites developed and amongst Web authors.


Two target populations will be studied and approximately 15 people from each will be interviewed:


As these are likely to be busy individuals, a brief interview schedule, lasting no more than 20 minutes will be developed, which can be administered by phone or face-to-face. The interview schedule will cover:


Data will be analysed to determine the level of knowledge of participants of accessible design; to uncover any areas where lack of awareness or understanding is particularly apparent; to determine what information and tools Web authors would like to support them in developing accessible Web material.

Deliverable: Report on knowledge of and attitudes towards Web accessibility

Study 2: Usability of the Web Content Accessibility Guidelines by professional Web authors

This study will undertake a user-based evaluation of the Guidelines with Web authors.


Two target populations will be studied and approximately 12 people from each will be included in the study:

It is thought important to include these two populations, as both types of web authors are very active in producing material for the web and the WCAG guidelines need to be useful and effective for both populations.


The methodology will be similar to that used by Colwell and Petrie (1999) which was found to be effective and yielded much useful information.

Where possible the study will be conducted in participants' normal web authoring environment - their office or home. Permission will be sought to video the session for later detailed analysis.

The study will be divided into 3 phrases of data collection, and will require two or three sessions with a researcher.

Phase 1: Baseline knowledge of Web accessibility

Participants will be interviewed about their knowledge of accessibility issues; they will then be shown a number of carefully chosen Web sites and asked to identify accessibility problems.

Phase 2: Understanding the Guidelines

Participants will be asked to study the Guidelines in their own time. They will then be re-interviewed about accessibility issues and shown another set of Web sites and asked to identify accessibility problems.

This phase will test how easily they can absorb information about Web accessibility from the Guidelines.

Phase 3: Applying the Guidelines

Participants will be provided with some content material for a set of Web pages and asked to turn this into an accessible Web site. The material will be carefully chosen so that it provides a range of challenges in terms of accessibility.

Participants will be asked to provide a "concurrent think aloud protocol", that is to provide a (brief) running commentary on what they are doing and what problems they are facing. The researcher will sit with the participant, observing the authoring process and prompting them where necessary to describe their actions. This technique has been shown in many studies to provide rich data of the participants understanding of the situation.


Data will be analysed to determine:

Deliverables: Report on the evaluation, plus recommendations for, where appropriate, improving the usability of the WCAG


To be agreed with the WAI.

It is estimated that the project would take 12 months to complete.

Costing and Resources

An application has been submitted to the Nuffield Foundation for funding of the project.

Researchers will come from both the University of Dundee and University of Hertfordshire.

Further details of costing and resources have yet to be finalised.


  1. Colwell, C. and Petrie, H. (1999). A preliminary evaluation of the WAI guidelines for producing accessible web pages. In C. Bühler and H. Knops (Eds.), Assistive technology on the threshold of the new millennium. Amsterdam: IOS Press.
  2. De Souza, F, and Bevan, N (1990) The use of guidelines in menu interface design: evaluation of a draft standard. In Diaper, D., Gilmore, D., Cockton, G. and Shackel, B. (eds) Human Computer interaction - INTERACT '90, pp435-440.
  3. Mosier, J. and Smith, S. (1987) Application of guidelines for designing user interface software. Behaviour and Information Technology, Vol. 5, No. 1 pp39-46.

Please email comments/feedback to David Sloan:

Back to start of proposal

Valid XHTML 1.0!