digital media access group
...excellent accessibility research and consultancy
This is a archived version of the DMAG website, but the information remains for reference. Please visit the new website for updated information.
Testing with disabled users - is it worth the effort?
By David Sloan, published 20th April 2004.
Should I test with disabled people?
The issue of whether or not testing a web site with disabled people is good practice, or perhaps effectively required by legislation has become a hot topic.
In a recently published (14th April 2004) study [1] into the accessibility of UK web sites, commissioned by the Disability Rights Commission (DRC) and carried out by experts at City University London, a key conclusion of the report was that some problems found during the research could not have been avoided by developers following guidelines, and only through user testing with disabled people. From this, a key recommendation was made that disabled people be involved throughout the development lifecycle of a web site.
Immediate reaction to the report by the W3C [2] indicated that there's some uncertainty over the relationship between the Web Content Accessibility Guidelines [3], accessibility of web sites to disabled people, and usability of web sites to disabled people.
Yet, looking at the legislation of the UK's Disability Discrimination Act (DDA)[4], one obligation of service providers is to ensure that not only disabled people are able to access a service, but that they do not unjustifiably receive a poorer service. This could be interpreted as meaning that, for web sites, usability for disabled people may be a legislative obligation under the DDA.
The DRC recommendations shouldn't be surprising - after all they are simply restating the fundamental principles of user-centred design, where end users are involved in evaluation of a product, not just at the end of the design lifecycle, but from the start and throughout. So in an ideal world, in order to ensure optimal accessibility and usability, every web site should be evaluated with a number of disabled people.
Problems
But some valid points have been raised (for example in [5]) about this ideal:
- 1. Disabled people are individuals - even within what might be seen as a homogenous group, like 'blind people', there will be large variations in technical knowledge, past experience in web usage, and in expectations, attitudes and tolerance levels. And remember that, statistically, most people who face access barriers do so as a result of a combination of relatively minor impairments that combine to present more significant problems - yet who generally do not use, or need to use, assistive devices.
- 2. Browsing technology available may also affect accessibility - the W3C's response stresses that the accessibility of a web site depends on more than just the design of the site itself, but also the ability of the browser and any assistive technology used to output the site. So a site may be adequately accessible to someone with an expensive browsing set-up, but cause horrendous problems for someone using older, less capable assistive technology.
- 3. Finding even one person with a specific impairment to test a site can be difficult and time consuming for some site developers - though as the profile of web accessibility is raised, this may become less of an issue. There are email discussion groups e.g. [6] where disabled people can be contacted and requests made for help, but even then, this may introduce a bias towards 'expert disabled web evaluators'.
Guidelines are for technical accessibility
Given the issues above, you might think that finding a couple of expert blind web users to test your web site is a waste of time. You may reasonably argue that the findings of one or two people with specific access needs may not be representative of an overall population - and to act on these findings may result in a site optimised for expert users of the latest assistive technology users, but very difficult for other similarly impaired people with a different browsing set-up.
It's been argued that that these issues are why we have guidelines such as the WCAG. They are, after all, produced from the findings of an extensive consultative exercise involving industry and academic experts and disabled web users - distilling these findings into a set of guidelines that developers can follow to avoid problems and produce sites that as many people as possible can access with as little difficulty as possible. This saves a hard-pressed developer the time and money required to recruit and find people who use a specific assistive technology.
So if you're of the opinion that testing your web site with disabled people is all about technical accessibility - finding out whether a specific assistive technology can work with your site, then you may become frustrated to discover just how many browsing set-ups there are, and decide not to bother. Yet, with thorough research into browser and assistive technology conformance with the User Agent Accessibility Guidelines (UAAG) [7], a developer could know at a glance how well a particular browser or assistive technology/browser combination would cope with a web site that has been designed to follow guidelines, and make a decision on whether a change in design or work-around may be necessary.
This may be ambitious, but something similar exists in the form of Eric Meyer's CSS Support Chart [8]; and, encouragingly, Gez Lemon has started on an assistive technologies chart at [9] - so perhaps a thorough and rigorous table of conformance is not so far off. Any such chart would need to acknowledge that for many people with minor impairments, a graphical browser should be quite capable of being their assistive technology!
Until then, you'll need to continue to test the site yourself using some demo versions of popular assistive technologies, and 'disabling browsers' such as Lynx. Remember though, that this is best suited to finding technical accessibility issues. Don't assume that your experience as an occasional, non-disabled user of an assistive technology can be equated to that of a regular - or novice - disabled user.
Disabled people as customers, not human accessibility checkers
But why not think about disabled people as site users, as potential customers, and not just as human versions of Bobby or Usablenet's Lift? Instead of concentrating on whether their specific browsing set-up and impairment prevents them from using the site, think about usability.
Involve disabled people as usability evaluators. Find out how well can they complete the tasks they intended to carry out with the site? What problems did they find? What could be improved for them? Would they use the site again? Did they enjoy using the site? Maybe this feedback is harder to translate to design fixes than the results of a Bobby check; may not easily translate into unfulfilled WCAG checkpoints. But if acted on, it will produce a positive impact on the success of the resource.
DMAG has learned much from watching people with a wide range of impairments use web sites. We've also seen the reaction of technically very competent developers who, confident they had followed accessibility guidelines during development, watched in astonishment as their site caused serious problems for people with specific access impairments during user testing.
People are different, and that includes disabled people. So instead of worrying about the effectiveness of an accessibility evaluation with one blind 'geek', involve them, and as many people you can find with impairments, however mild, in usability testing. As well as uncovering potential barriers to use, this will help you get to know how people with specific impairments approach web browsing.
Implementing an evaluation and testing programme that separates technical accessibility issues from usability issues, that may or may not be exaggerated by the presence of a specific impairment, is an anticipatory action towards limiting the likelihood of disabled people receiving an unjustifiably poorer service. It'll also help you towards a site that is not just technically accessible but is easy, efficient and enjoyable to use by as many people as possible.
References
- 1. DRC Formal Investigation into Web Accessibility:
- http://www.drc.org.uk/publicationsandreports/report.asp
- 2. W3C Web Accessibility Initiative Statement on DRC Investigation:
- http://www.w3.org/2004/04/wai-drc-statement.html
- 3. W3C Web Content Accessibility Guidelines (WCAG):
- http://www.w3.org/TR/WCAG10/
- 4. Disability Discrimination Act (DDA) 1995:
- http://www.legislation.hmso.gov.uk/acts/acts1995/Ukpga_19950050_en_1.htm
- 4. A thread from the W3C's WAI-IG email discussion group, on the implications of the DRC report:
- http://lists.w3.org/Archives/Public/w3c-wai-ig/2004AprJun/0100.html
- 5. Usability for Visually Impaired persons List (UVIP):
- http://www.usabilitynews.com/news/article970.asp
- 6. W3C User Agent Accessibility Guidelines (UAAG):
- http://www.w3.org/TR/UAAG10/
- 7. CSS Support Chart:
- http://devedge.netscape.com/library/xref/2003/css-support/
- 8. Gez Lemon's assistive technology conformance chart:
- http://www.juicystudio.com/assistivedeviceschart.asp