USER INTERFACE RATING FORM


Click here to download a Microsoft Word Rich Text Format (rtf) version of the User Interface Rating Form.

Instructions For Use

 

  1. The "User Interface Rating Form" should be used by experienced interactive multimedia designers or users to rate the interface of a new program or one under development.
  2. The ten criteria used in the "User Interface Rating Form" are explained in detail at the end of the instrument itself.
  3. Some criteria may not be relevant in every IMM program.


User Interface Rating Tool for Interactive Multimedia

©1993 Thomas C. Reeves, Ph.D. and Stephen W. Harmon, Ed.D.

Instructions: For each of ten user interface dimensions illustrated below, rate the program you have reviewed on a one to ten scale by circling the appropriate number under the dimension. (Accompanying this tool are definitions for each of the ten user interface dimensions.) Please add any comments that may help to clarify or explain your rating. If a specific dimension does not seem appropriate to the interactive program you are reviewing, do not circle any numbers on the scale for that dimension and add a brief comment to explain your response.


Comments:



Comments:



Comments:



Comments:



Comments:



Comments:



Comments:



Comments:



Comments:



Comments:


Please add other comments related to the user interface of this program below:




Definitions for User Interface Rating Tool

User Interface Dimension 1 - Ease of Use

"Ease of Use" is concerned with the perceived facility with which a user interacts with an interactive multimedia program. Figure 1 illustrates a dimension of such a program ranging from the perception that the program is very difficult to use to one that is perceived as being very easy to use. Like many of the dimensions described in this tool, ease of use is both an aggregate and individual dimension. For example, in the aggregate sense, the Windows interface is generally perceived as easier to use than the command interface of the Microsoft disk operating system (MS DOS). However, in the individual sense, some people may perceive the MS-DOS interface to be easier to use because of their own unique experiences and attributes.

Figure 1. "Ease of Use" dimension of user interface.

User interface dimensions may be highly correlated with how well users enjoy using a specific program. Whether users like a program may be more or less important, depending on the intent of the program and the context for its use. Certainly, not liking an interactive program that is intended to be highly motivating is a major problem, whereas users' affect for a program may be less important in a training context in which strong extrinsic motivational factors exist. Nonetheless, in the long run, improving the user interface dimensions of multimedia, such as "ease of use," is a highly desirable goal, regardless of context.

User Interface Dimension 2 - Navigation

"Navigation" is concerned with the perceived ability to move through the contents of an interactive program in an intentional manner. Figure 2 illustrates a dimension of interactive multimedia ranging from the perception that a program is difficult to navigate to one that is perceived as being easy to navigate. An important aspect of navigation is orientation, i.e., the degree to which a user feels that he/she knows where he/she is in a program and how to go to another part of it. This is a critical variable because users frequently complain of being lost in a interactive program (Utting & Yankelovitch, 1989). Designers use several ways of supporting navigation and maintaining orientation. A popular approach to navigation is the WIMP (window-icons-mouse-pointing) interface.

Figure 2. "Navigation" dimension of user interface.

User Interface Dimension 3 - Cognitive Load

Using an interactive program requires different mental efforts than performing tasks via print or other media. In order to make any meaningful response to an interactive program, users must cope with and integrate at least three cognitive loads or demands, i.e., (a) the content of the program, (b) its structure, and (c) the response options available. To use interactive programs, users must perceive options, conceptualize a choice, and make some physical action, all while mentally coordinating the demands of these three cognitive loads. The user interface is the vehicle that allows perceptual, conceptual, and physical contacts with the interactive program. In terms of "cognitive load," the user interface can seem unmanageable (i.e., confusing) at one end of the continuum and easily manageable (i.e., intuitive) at the other end (see Figure 3).

Figure 3. "Cognitive Load" dimension of user interface.

Learners acquire and structure information delivered via interfaces, conduct mental operations, and accomplish physical activities during their interactions with interactive multimedia. The limited capacity of working memory to hold only five to nine chunks of information simultaneously makes it difficult for users of complexity structured programs to reason when numerous cognitive load factors must be handled simultaneously. Users may feel overwhelmed by numerous options that increase the cognitive load. The risks of confusion are especially high when users confront programs which by their very nature include many interactive options. The possibility of user disorientation is a major concern in the increasingly popular multimedia programs that feature a complex, flexible structure.

User Interface Dimension 4 - Mapping

"Mapping" refers to the program's ability to track and graphically represent to the user his or her path through the program. In complex, non-linear programs, user-disorientation can be alleviated if users can see what parts of the system they have already accessed. Utting and Yankelovitch (1989) discuss user disorientation as referring to, among other things, the user's not knowing "the boundaries of the information space." Having a detailed mapping system gives users an aid in understanding which parts and how much of the information space they have interacted with, and conversely which parts and how much of it they haven't. Interactive programs fall in a continuum of containing no mapping function to an appropriately powerful mapping function (see Figure 4).

Figure 4. "Mapping" dimension of user interface.

The notion of an "appropriately powerful" mapping function requires some explanation. Just as it is important to possess a map of the most usable scale when taking a road trip, it is important for interactive programs to provide enough, but not too much, detail in showing user paths. A map that shows every piece of a program's knowledge space might prove to be so tedious or unwieldy as to be of as little value as an interactive program with no map.

User Interface Dimension 5 - Screen Design

"Screen Design" is a particularly complex dimension of interactive programs that can easily be broken down into many sub-dimensions related to text, icons, graphics, color, and other visual aspects of interactive programs. Shneiderman (1987) maintains that although certain design principles have been established, "screen design will always have elements of art and require invention" (p. 326). A separate dimension has been defined to deal with the artistic aspects of interactive programs (see Dimension 9 - Aesthetics below). We define "screen design" as a dimension ranging from substantial violations of principles of screen design to general adherence to principles of screen design (see Figure 5).

Figure 5. "Screen Design" dimension of user interface.

There are two problems with this dimension. First, screen design principles have not kept up with the rapidly changing nature of interactive technology. Second, creative designers may sometimes intentionally violate screen design principles for effect or to otherwise focus the user's attention. Nonetheless, we think that there exists enough knowledge about the principles of screen design that people, particularly experienced designers, can make meaningful distinctions among poorly and well designed screens in interactive programs.

User Interface Dimension 6 - Knowledge Space Compatibility

"Knowledge space" refers to the network of concepts and relationships that compose the mental schema a user possesses about a given phenomena, topic or process. Subject matter experts and/or designers of interactive programs are generally perceived as possessing an expert knowledge space with respect to the content included in the programs they create. This expertise usually is the basis for the structure of the knowledge or information presented in a program. Novice users, on the other hand, often possess an inadequate knowledge space with respect to the content of a program. The knowledge space of novices may be inadequate because of ignorance, misconceptions, or some blending of ignorance and misconceptions. When a novice user initiates a search for information in an interactive program, the interface should be powerful enough so that the user perceives the resulting information as compatible with his or her current knowledge space (see Figure 6). If the information received is not perceived as relevant to the search strategies used by the user, the system will be perceived as incompatible.

Figure 6. "Knowledge Space Compatibility" dimension of user interface.

Admittedly, this is a very difficult dimension to judge. However, if a user initiates a search for information about a topic, e.g., the procedures for installing new software, the resulting information should seem compatible with that search once the information is thoroughly explored. If the information seems arbitrary or irrelevant to the search that was initiated, the knowledge space representation should be judged as incompatible.

User Interface Dimension 7 - Information Presentation

The "Information Presentation" dimension is concerned with whether the information contained in the knowledge space of an interactive program is presented in an understandable form. The most elegantly designed user interface for an interactive program is useless if the information it is intended to present is incomprehensible to the user. Certainly the user might be able to find all of the information about a subject, but whether the user could then comprehend, understand, or learn that information is another matter. Imagine a complicated installation procedure presented in textual form, written in a stream of consciousness style reminiscent of James Joyce's Ulysses. Or consider a video presentation on sales techniques for ATMs, directed and produced by Andy Warhol. In each case the information requisite for understanding may be present, but would probably be difficult if not impossible to comprehend. Information presentation is defined as a dimension ranging from obtuse to clear (see Figure 7).

Figure 7. "Information Presentation" dimension of user interface.

User Interface Dimension 8 - Media Integration

The most important aspect of the media integration dimension refers to how well an interactive program combines different media to produce an effective whole. Do the various media (text, graphics, audio, video, etc.) work together to form one cohesive program, or is the program a hodgepodge of gratuitous media segments? Are the various media components necessary to the function of the program or would the program function equally as well without them? The media integration dimension is defined as ranging from uncoordinated to coordinated (see Figure 8).

Figure 8. "Media Integration" dimension of user interface.

User Interface Dimension 9 - Aesthetics

"Aesthetics" refers to the artistic aspects of interactive programs in the sense of possessing beauty or elegance. In the aggregate sense, many people may praise the aesthetics of an automobile design or the elegance of a bridal gown. However, in an individual sense, aesthetics are highly unique and one person's sense of the beautiful may seem grotesque to another. Eisner (1985) described the need to develop "connoisseurs" in evaluation of education and training, just as we have connoisseurs in the arts. Connoisseurs have refined tastes and a deep sensitivity to aesthetics that enable them to criticize phenomena (e.g., plays, paintings, musical scores, or interactive programs) in a manner that can be communicated to others. In turn, other people, perhaps less refined and less sensitive, may become more informed by "consuming" the expert reviews provided by the connoisseurs. In the absence of such connoisseurs, the aesthetics dimension of the user interface of an interactive multimedia program is defined as ranging from displeasing to pleasing (see Figure 9).

Figure 9. "Aesthetics" dimension of user interface.

User Interface Dimension 10 - Overall Functionality

"Overall Functionality" is an aspect of interactive multimedia programs related to the perceived utility of the program. The perceived functionality of an interactive program is obviously highly related to the intended use of the program. A given program may have multiple uses. Its overall functionality must be judged in relation to the specific intended use that exists in the mind of the users. Figure 10 illustrates a dimension of the user interface of interactive programs that ranges from dysfunctional to highly functional.

Figure 10. "Overall Functionality" dimension of user interface.