Heuristic Evaluation Instrument and
Protocol for E-Learning Programs

Draft of September 5, 2001

(This instrument was created Lisa Benson, Dean Elliott, Michael Grant, Doug Holschuh, Beaumie Kim, Hyeonjin Kim, Erick Lauber, Sebastian Loh, and Tom Reeves from The University of Georgia.)

Click here to download a Microsoft Word Rich Text Format (rtf) version of the Heuristic Evaluation Instrument.

Introduction:

This instrument and protocol are intended for use by instructional designers and other experts who are engaged in heuristic evaluations of e-learning programs. The instrument itself lists twenty heuristics for e-learning programs, some of which are based upon Jakob Nielsenís widely used protocol for heuristic evaluation of any type of software (http://useit.com/papers/heuristic/), and the rest of which are based upon factors related to instructional design. Although we have tried to be comprehensive, experts may decide to add new heuristics deemed relevant to the types of e-learning programs being evaluated or to the expertís specific expertise.

Protocol:

1. An expert should review the heuristics and accompanying ìSample questions to ask yourselfî in the instrument before reviewing an e-learning product. The expert should modify the instrument if needed, by adding, deleting, or changing heuristics.

2. It is recommended that the expert spend sufficient time exploring the e-learning program before beginning the actual heuristic evaluation. Ideally, the expert will assume the role of typical learner who would use this e-learning program. Before beginning the review, the expert should be given (or try to discover) background information related to the e-learning program such as:

a. Target audience and learner characteristics: A thorough description of the intended audience and their learner characteristics (e.g., education level, motivation, incentive, and computer expertise) will enable the expert to judge the appropriateness of the user interface and other aspects of the programís usability in an informed manner.

b. Instructional goals and objectives: The expert should know as much as possible about the needs that the e-learning program is intended to address, ideally in terms of clear goals and objectives.

c. Typical context for using this program: Realistic scenarios for when, where, and how the e-learning program will be used should be described to the expert.

d. Instructional design strategies used in the program: If possible, a description of the design specifications used in developing the e-learning program should be provided to the expert so that the expertís judgment of the appropriateness of the instructional design strategies are informed with respect to the instructional designerís intentions.

e. The status of the programís development and possibilities for change: The expert should be informed as to where the program is in the development cycle (e.g., an early prototype, a beta version, or a completed version under consideration for redesign).

3. After spending enough time to become familiar with the program, the expert should go through it from beginning to end to conduct the actual heuristic evaluation. (With very long programs, the expert may only go through a representative sample of the program.)

4. The expert should make note of every usability problem found. For each problem, the expert should identify the heuristic it violates, and then give it a severity rating using the severity scale below. If the problem cannot be attributed to a violation of a specific heuristic, the expert should make a note of this. (If a number of problems are found that cannot be associated with specific heuristics, this may suggest the need for the development of new heuristics.)

Severity Scale

1) cosmetic problem only; need not be fixed unless extra time is available

2) minor usability problem; fixing this should be given low priority

3) major usability problem; important to fix; so should be given a high priority

4) usability catastrophe; imperative to fix before this product is released

5. After all the usability problems are found, the expert should go back though them and give each one an extensiveness rating using the extensiveness scale below.

Extensiveness Scale

1) this is a single case

2) this problem occurs in several places in the program

3) this problem is widespread throughout the program

6. Most heuristic evaluations involve 4 or 5 experts. Once all the experts have completed their evaluations, they may be brought together for a debriefing led by a moderator. The discussion of the usability problems may be videotaped for further analysis. If major differences appear in the problems found or the ratings given, the moderator may try to get the experts to resolve their differences and reach consensus. The experts may also be asked to suggest strategies for resolving the major usability problems they found.

7. A heuristic evaluation report should then be compiled. Bar charts, tables, and other illustrations should be used to display the results. Screen captures can also be incorporated into the report to illustrate major problems and suggested enhancements.

8. The most important component of the heuristic report is a set of recommendations for improving the usability of the e-learning program.These should be as specific as possible to provide the designers with the information they need to eliminate the problems and improve the e-learning program.


 

Usability Heuristics

1. Visibility of system status: The e-learning program keeps the learner informed about what is happening, through appropriate feedback within reasonable time.

Sample questions to ask yourself:

a. Does the learner know where they are at all times, how they got there, and how to get back to the point from which they started?

b. When modules and other components of the e-learning (e.g., streaming video) are loading, is the status of the upload communicated clearly?

c. Does the learner have confidence that the e-learning program is operating the way it was designed to operate?

Additional comments:

 

2. Match between system and the real world: The e-learning programís interface employs words, phrases and concepts familiar to the learner, rather than system-oriented terms. Wherever possible, the e-learning program utilizes real-world conventions that make information appear in a natural and logical order.

Sample questions to ask yourself:

a. Does the e-learning programís navigation and interactive design utilize metaphors that are familiar to the learner either in terms of traditional learning environments (e.g., lectures, quizzes, etc.) or in terms related to the specific content of the program?

b. Is the cognitive load of the interface as low as possible to enable learners to engage with the content, tasks, and problems as quickly as possible?

c. Does the e-learning program adhere to good principles of human information processing?

Additional comments:

 

3. User control and freedom: The e-learning program allows the learner to recover from input mistakes and provides a clearly marked ìemergency exitî to leave an unwanted state without having to go through an extended dialogue.

Sample questions to ask yourself:

a. Does the e-learning program allow the learner to move around in the program in an unambiguous manner, including the capability to go back and review previous sections?

b. Does the e-learning program allow the learner to leave whenever desired, but easily return to the closest logical point in the program?

c. Does the e-learning program distinguish between input errors and cognitive errors, allowing easy recovery from the former always, and from the latter when it is pedagogically appropriate?

Additional comments:

 

4. Consistency and standards: The e-learning program is consistent in its use of different words, situations, or actions and it adheres to general software and platform conventions.

Sample questions to ask yourself:

a. Does the e-learning program function properly as long as the computerís screen resolution, memory allocations, bandwidth, browsers, plug-ins, and other technical aspects meet the required specifications?

b. Does the e-learning program include interactions that are counter-intuitive with respect to common software conventions?

c. Does the e-learning product adhere to widely recognized standards for interactions (e.g., going back in a Web browser)?

Additional comments:

 

5. Error prevention: The e-learning program is carefully designed to prevent common problems from occurring in the first place.

Sample questions to ask yourself:

a. Is the e-learning program designed so that the learner recognizes when he/she has made a mistake related to input rather than content?

b. Is the e-learning program designed to take advantage of screen design conventions and guidelines that clarify meaning?

c. Is the e-learning program designed to provide a second chance when unexpected input is received (e.g., ìYou typed ìbatî in response to the question. Did you mean ìtab?î)?

Additional comments:

 

6. Recognition rather than recall: The e-learning program makes objects, actions, and options visible so that the user does not have to remember information from one part of the program to another. Instructions for use of the program are visible or easily retrievable.

Sample questions to ask yourself:

a. Does the interface of the e-learning program speak for itself so that extensive consultation of a manual or other documentation does not interfere with learning?

b. Are icons and other screen elements designed so that they are as intuitive as possible?

c. Does the e-learning program provide user-friendly hints and/or clear directions when the learner requests assistance?

Additional comments:

 

7. Flexibility and efficiency of use: The e-learning program is designed to speed up interactions for the experienced learner, but also cater to the needs of the inexperienced learner.

Sample questions to ask yourself:

a. Is the e-learning program designed to make the best use of useful graphics and other media elements that download as quickly as possible?

b. Is the e-learning program designed to allow large media files to be downloaded in advance so that learner wait time is minimized?

c. Does the program allow keyboard short cuts that make frequent interactions as efficient as possible?

Additional comments:

 

 

8. Aesthetic and minimalist design: Screen displays do not contain information that is irrelevant, and ìbells and whistlesî are not gratuitously added to the e-learning program.

Sample questions to ask yourself:

a. Are the font choices, colors, and sizes consistent with good screen design recommendations for e-learning programs?

b. Are extra media features (e.g., streaming video) in the e-learning program supportive of learning, motivation, content, or other goals?

c. Does the e-learning program utilize white space and other screen design conventions appropriately?

Additional comments:

 

9. Help users recognize, diagnose, and recover from errors: The e-learning program expresses error messages in plain language (without programmer codes), precisely indicates the problem, and constructively suggests a solution.

Sample questions to ask yourself:

a. Does the learner receive meaningful feedback concerning the nature of any input they make into the program?

b. If the learner answers a question incorrectly, is he/she told the correct answer and why the answer given was wrong, if this is instructionally appropriate?

c. When feedback is provided, is it given in a clear, direct, and friendly (non-condescending) manner?

Additional comments:

 

10. Help and documentation: When it is absolutely necessary to provide help and documentation, the e-learning program provides any such information in a manner that is easy to search. Any help provided is focused on the learner's task, lists concrete steps to be carried out, and is not be too large.

Sample questions to ask yourself:

a. Is help provided that is screen or context specific?

b. Is help and documentation available from any logical part of the e-learning program?

c. Does the e-learning program include a map or table of contents that allows you to see what you have seen and not seen?

Additional comments:

 

 

11. Interactivity: The e-learning program provides content-related interactions and tasks that support meaningful learning.

Sample questions to ask yourself:

a. Does the e-learning program provide too many long sections of text to read without meaningful interactions?

b. Does the e-learning engage the learner in content-specific tasks to complete and problems to solve that take advantage of the state-of-the-art of e-learning design?

c. Does the e-learning program provide a level of experiential learning congruent with the content and capabilities of the target audience?

Additional comments:

 

 

12. Message Design: The e-learning program presents information in accord with sound principles of information-processing theory.

Sample questions to ask yourself:

a. Is the most important information on the screen placed in the areas most likely to attract the learnerís attention?

b. Does the e-learning program follow good information presentation guidelines with respect to organization and layout?

c. Are graphics in the e-learning program used to clarify content, motivate, or serve other pedagogical goals?

Additional comments:

 

 

13. Learning Design: The interactions in the e-learning program have been designed in accord with sound principles of learning theory.

Sample questions to ask yourself:

a. Does the e-learning program provide for instructional interactions that reflect sound learning theory?

b. Does the e-learning program engage learners in tasks that are closely aligned with the learning goals and objectives?

c. Does the e-learning program inform learners of the objectives of the program and remind them of prior learning?

Additional comments:

 

 

14. Assessment: The e-learning program provides assessment opportunities that are aligned with the program objectives and content.

Sample questions to ask yourself:

a. Does the e-learning program provide opportunities for self-assessments that advance learner achievement?

b. If appropriate to the context, do assessments provide sufficient feedback to the learner to provide remedial directions?

c. Are higher order assessments (e.g., analysis, synthesis, and evaluation) provided wherever appropriate rather than lower order assessments (e.g., recall and recognition)?

 

Additional comments:

 

15. Media Integration: The inclusion of media in the e-learning program serves clear pedagogical and/or motivational purposes.

Sample questions to ask yourself:

a. Is media included that is obviously superfluous, i.e., lacking a strong connection to the objectives and design of the program?

b. Is the most appropriate media selected to match message design guidelines or to support specific instructional design principles?

c. If appropriate to the context, are various forms media included for remediation and/or enrichment?

Additional comments:

 

 

16. Resources: The e-learning program provides access to all the resources necessary to support effective learning.

Sample questions to ask yourself:

a. Does the e-learning program provide access to a range of resources (e.g., examples or real data archives) appropriate to the learning context?

b. If the e-learning program includes links to external World Wide Web or Intranet resources, are the links kept up-to-date?

c. Are resources provided in a manner that replicates as closely as possible their availability and use in the real world?

 

Additional comments:

 

 

17. Performance Support Tools: The e-learning program provides access to performance support tools that are relevant to the content and objectives.

Sample questions to ask yourself:

a. Are performance support tools provided that mimic their access in the real world?

b. Provided the context is appropriate, does the e-learning program provide sufficient search capabilities?

c. Provided the context is appropriate, does the e-learning program provide access to peers, experts, instructors, and other human resources?

Additional comments:

 

18. Learning Management: The e-learning program enables learners to monitor their progress through the material.

Sample questions to ask yourself:

a. Does the learner know what he/she is doing and how he/she is doing within various parts of the e-learning program?

b. Does the learner perceive options for additional guidance, instruction, or other forms of assistance when it is needed?

c. Does the learner possess an adequate understanding of what he/she has completed and what remains to be done within any specific unit (e.g., a course) of e-learning?

Additional comments:

 

19. Feedback: The e-learning program provides feedback that is contextual and relevant to the problem or task in which the learner is engaged.

Sample questions to ask yourself:

a. Is the feedback given at any specific time tailored to the content being studied, problem being solved, or task being completed by the learner?

b. Does feedback provide the learner with information concerning his/her current level of achievement within the program?

c. Does the e-learning program provide learners with opportunities to access extended feedback from instructors, experts, peers, or others through e-mail or other Internet communications?

Additional comments:

 

20. Content: The content of the e-learning program is organized in a manner than is clear to the learner.

Sample questions to ask yourself:

a. Is the content organized in manageable modules or other types of units?

b. Is the content broken to appropriate chunks so that learners can process them without too much cognitive load?

c. Does the e-learning program provides advanced organizers, summaries, and other components that foster more efficient and effective learning?

 

Additional comments:

 

 

NOTE: Experts should modify the heuristics noted above, or delete or add others, as needed for the specific type of e-learning program that is being evaluated.