Project, Part 3

Due Sat May 5 in Canvas

Checkpoint 1: Your evaluation plan is due Sat, April 14, 11:59pm in Canvas. Feedback will be returned to your group by Wed, Apr 18.

Goals

The goals of this phase of the project are to provide you with experience:

  • Evaluating your designs and performing evaluations of others' designs.
  • Refining the design to develop a final, high fidelity prototype.

Details

In this project phase, your goal is to perform a set of evaluations on your design to understand its strengths and weaknesses. The findings from these evaluations will feed directly into a final, high fidelity prototype of your design.

No design is perfect; any design can always be improved. Your evaluations will indicate what is both good and bad about your design. Accept the fact that you will need to "let go" of some design ideas you liked and that you will need to iterate to better suit your users' needs. Be open to what needs to be changed about your design and don't be defensive -- remember, you are trying to make computational systems that people find useful and usable, not convince others that your design is superior. We expect to see changes -- both large and small -- in your final system that arise from the findings of your evaluations.

You will need to perform evaluations with two groups of people: Your actual end users and experts in the field of HCI. For this class, the experts are other teams in the class -- you will evaluate each others' projects using techniques such as Heuristic Evaluation and Cognitive Walkthrough. We will assign 2 teams to work as evaluators for each project.

Once you have completed your evaluations and iterated on the design, you will need to create a final, high fidelity prototype. The final, high fidelity representation should be sufficiently detailed that it is indistinguishable from an actual, functioning product. You may "fake" computation by employing "hard-wired" (canned) data and computation, but otherwise, the final prototype should mimic a real system as much as possible. If your design is heavily based in the physical world and you are uncertain how to proceed, set up an appointment with the professor or come in during office hours.

Planning the Evaluations

You will need to develop and follow an evaluation plan for your designs. There is no single best way to evaluate a design, so you MUST use multiple techniques/methods to assess the design. Use at least 3 techniques. You have the following techniques at your disposal:

  • Naturalistic methods (interviews, observations of use) using WoZ techniques.
  • Heuristic evaluation.
  • Cognitive walkthrough.
  • Think aloud.
  • Custom evaluation methods (you will need to get feedback from the professor if you develop your own evaluation techniques).

Remember, you will need to evaluate your design with your actual users and with two other project groups, who will act as expert evaluators. We will make pairings between groups when we enter this phase of the project.

Coordinate with each other so each group has the opportunity to evaluate each others' designs.

Performing Expert Evaluations: Guidelines

It is extremely easy to find fault with others' work. It is much harder to find the strengths and to provide constructive criticism, in a professional manner, that serves to improve the project rather than tear it down. As you evaluate each others' work, try to follow these guidelines:

  • Be professional.
  • Be sure you completely understand who the users are, their tasks, and the problems that need to be addressed.
  • Suggest improvements rather than point out flaws. Do so by asking rather than telling or commanding. For example, "What if you did this?" Or, "Have you thought about trying this approach?"
  • Note what you like about the design.
  • Listen carefully. Try to understand their reasoning for the design.
  • Don't press a point. Even if you think you have found a major design flaw, clearly articulate your point and the reasons why you feel the way you do, but do not try to win them over to your point of view.
  • Provide concrete reasoning to back up your suggestions.
  • People can easily become defensive about their work. Be sensitive to this and back off or move on to other issues if discussion becomes a bit heated or argumentative.

Deliverables

Video Walkthrough of Final High-Fidelity Prototype

You must create a video of your final prototype. The format must be an mp4 file. The video must be submitted via Canvas. The video should be 5-7 min long. It cannot be longer than 10 min.

Written Report

You must submit a final report that includes the following sections:

  • Introduction
    • A summary of your users, their tasks, their current practices, and the strengths and limitations of current practices.
    • What particular problem(s) you decided to address.
    • An overview of your design.
    • An overview of the evaluation methods used.
    • An overview of the most significant findings of your evaluation.
    • The introduction should serve as a brief summary of the entire project.
  • Evaluation Plan and Discussion
    • Provide the final, detailed evaluation plan used to evaluate your designs.
    • Provide a summary of what actually transpired: How many subjects participated? How long did the evaluations take? What went smoothly, what could have gone better?
  • Evaluation Results
    • Present the major findings from conducting the evaluations. What did you learn? What surprised you? What worked, what didn't work in your design?
    • When discussing your findings, be sure to indicate what findings were found where (i.e., with actual end users vs. expert evaluators).
    • Discuss the ramifications of the findings -- what aspects of the design did you have to change? What couldn't you change, and why? Were the changes major, affecting large parts of the interaction design, or relatively minor in nature?
  • Final Design
    • In this section, you must present a final, detailed design of your system. There should be no low fidelity representations. Take pictures, take screenshots, create annotated storyboards with screenshots, create a video prototype... Use whatever narrative and illustrative devices you need to accurately, succinctly, and fluidly present your designs. You may mock up interactive versions of your system, but we will evaluate your final design based on the contents of this section. In evaluating your design, we will consider the following:
      • How well does it meet user needs?
      • How well does it mesh with the culture of practice?
      • How thorough and complete is the design?
      • How realistic is the design?
  • Future Work
    • By the time you write the report, you will have completed the end-to-end process of studying users, defining needs, designing a system, and evaluating this system. You are now somewhat of an expert on this particular user group and a specific portion of their work. Given your insights, what promising avenues are there for further improving their work? What problems of theirs need to be better solved? In what ways could your design be extended to address these needs? View this section as a way to pass your knowledge on to future designers so they can learn from all the work you have done, rather than starting new when faced with the need to design for this user group.

Grading

For this final portion of the project, we will consider the following :

  • Does your video thoroughly demonstrate your high-fidelity prototype?
  • Is the information requested above clearly communicated?
  • Do you have each of the required sections listed above?
  • Are we able to determine the full use of your system from the descriptions provided?
  • Have you planned for error handling/recovery?
  • Does your design honor the constraints of technology (e.g., font sizes given current display resolutions)?
  • Does your system mesh with current work practices?
  • Have you performed a complete, worthwhile set of evaluations? Have you uncovered significant issues in your design? Have you made significant findings?
  • Have you iterated on your design to address identified problems?
  • Are your descriptions clear, coherent, and efficiently present your findings, as evidenced by presentation style, proper grammar, and spelling?