Deadline: Thursday, December 10 (last class)
Points: This assignment is worth 70 points (about 16% of your team project, 7% overall)

Assignment Overview
This assignment should be read in conjunction with TA07 Final Interactive Prototype and TA09 Final Video. Together, TA07, this assignment (TA08) , and TA09 represent the culmination of work on your team project over the course of the semester. They also serve as a "final exam" of sorts as we will not be having a formal exam during our final exam slot (instead, we will be viewing your TA09 final videos).

While TA07 and TA09 focus on the specifications and expectations for your final interactive prototype and final video respectively, this assignment describes the usability test for your prototype and the required sections for your final report.

What To Do

  1. Begin by reading over TA07 Final Interactive Prototype and TA09 Final Video. Operationally, I suggest you assign "leaders" within your team to manage different components of TA07, TA08, and TA09 so that nothing falls through the cracks and to divide labor more effectively. While I expect that each individual member should contribute to all three assignments, I think having a specific leader will help organize the remaining tasks.

  2. Once you implement the three primary tasks for TA07, you must perform usability tests with three users. Ordinarily, this would be done with "target" users pulled from your expected user population; however, in this case, you can use members of this class who are not on your team (this is a rule change compared to the paper prototype test). It is an interesting experience being on the subject side of usability testing, and I'd like for class members to have a chance to have the experience. That said, I strongly encourage you to test with at least one non-classmate (and it's perfectly fine if none of your test subjects are classmates). If you would like to recruit from the other CMSC434 section, please contact Dr. Vibha Sazawal (vibha@umiacs.umd.edu). Each user testing session must be done with at least two experimenters present (one to conduct the experiment and one to record observations/take notes). Before you conduct any test sessions, you must...

  3. Write a brief study plan. The study plan should include introductory content that you will read to each participant when they arrive at the test session, a description of your three primary tasks, a description of the type of data you plan to collect, the interview/survey instruments you plan to use, and the informed consent form you will have participants sign. For example, you should collect demographic data on your participants, which is often done via pre-study surveys. You should also give your participants brief post-study surveys inquiring about the effectiveness of your design and their overall user satisfaction. You will include this study plan in your report appendix. To help you with this, refer to Lazar, Feng and Hochheiser, Research Methods in Human-Computer Interaction. The user testing here will have a similar feel to TA05--only this time, your users won't be interacting with paper prototypes but a fully autonomous interactive prototype. :)

  4. After you write your study plan, test it out on a member of your team. This pilot test should follow the same protocol that you use for your three external users (i.e., it is a dress rehearsal for the real thing--try to make it as real as possible). You will include a section on how you conducted this pilot test, the results (if any), and how you changed your study protocol as a consequence (if you did). See "The Report" below.

  5. Now you're ready for the real usability tests. At a high level, each team should roughly follow this protocol:
    1. Recruit participants. You will have to describe your recruitment methodology in your report.

    2. As with TA05, have participants sign an informed consent. Make sure you include a section about asking to "video record" the session. At the beginning of the user testing session, read the "Purpose of this Study" section of the consent form out loud to your participants. This should be done consistently for each participant. Then, give your participants a chance to read the entire consent form, ask questions, and, if they agree to participate, have them sign the form. If they do not agree to participate, simply wish them a nice day and recruit another participant (it can be slightly awkward but this happens!). If they do agree to participate, provide a copy of the form and take the signed copy for yourself.

    3. If the participant agrees to video recording of the session, turn on the video camera. Do your best to protect the anonymity of your participant. Setup the video camera behind them (e.g., over their shoulder) so that you can capture their arms/hands interacting with the actual screens of your application. As with TA05, you can use anything to perform the video recording from a smartphone to a more sophisticated DSLR camera.

    4. After the informed consent process, you can begin testing your final interactive prototype. You might want to start with a brief "pre-study survey" asking basic demographic question (e.g., age, profession, computer proficiency). You will test each task in order. You can either read the task description aloud or have the participant read it him/herself. While executing the task, ask the participant to "think-aloud" (recall the think-aloud strategy we talked about earlier in the semester). One experimenter should record notes about how the participant is using the prototype, the problems/successes encountered, and any comments made during the testing session, as well as observations about non-verbal cues such as body language, facial expressions, and utterances. The experimenters should also record quantitative data such as how long each task took and the number of "problems" the user experienced. Include the qualitative and quantitative data notes for each participant in the appendix.

    5. After the three primary tasks have been tested, provide your participant with a short paper survey (called a "post-study survey") with a few questions inquiring about their overall impressions of the application and any specific comments for improvement. It's good to include at least one open-ended question where they can make any comment they want. Again, scan in and include these responses in the report appendix.

  6. After user testing, analyze your collected data (e.g., demographic survey data, observational notes, quantitative recordings, and post-study survey data). Your report should include a participant table describing each participant (but not using their name--to preserve anonymity, you can refer to them by initials or first name and last initial, or as participant 1, etc.) as well as a task performance table (rows are participants, columns are task timings and "problems" experienced for each task). Focus your analysis on the best/worst performing design elements and think about how these findings can be used to improve future design revisions. See Section 10.4.8 in Lazar, Feng and Hochheiser, Research Methods in Human-Computer Interaction for more information.

The Report

Submit a report of no more than 5 pages of text (approximately 2,500-3,500 words in a font similar to 11 pt. Times). Images, tables, figures, etc. are strongly encouraged, do not count against the page limit, and are thus effectively free.

Please construct your report with the following full sections and start each section with the header below (style Headings 1); it makes it easier for us to grade. So, for example, your first section should have the heading 1. Abstract and the next 2. Prototype Implementation.

Title Page

On your title page, please include a title (centered and bold), a catchphrase for your project that highlights its primary benefit, and your team name. Then make a table that lists each team member and describes their primary role/accomplishments for each of TA07, TA08, and TA09. If there's room, you may want to include some iconography or a screenshot of your app on the title page. Note: each team needs to assign an official proofreader that looks over the entire report, makes sure all of the requisite content is included, and eliminates typos/grammatical errors. Please include in the report who was assigned this role.

1. Abstract

The abstract should provide a 5-7 sentence overview of your report including: (i) a description of your application, (ii) a description of your three primary tasks, (iii) a description of your evaluation method, (iv) a summary of your primary findings, and (v) design implications for future revisions.

2. Prototype Implementation and Video

Describe your prototype implementation (TA07): what language did you use? Why? How did you build it? What libraries, if any, did you use? How were your design decisions informed? Did you rapidly iterate and test as you built the system? What interactions/features did you leave out but hoped to build? Justify your design decisions based on course content (e.g., lectures, readings, discussions).

Briefly discuss your video-making process (TA09). I realize that your video may not be completed when you turn in this report, but you can tell me what equipment and software you used or plan to use, how you settled on the idea, who worked on what, and so forth.

3. Task Descriptions

Describe your three primary tasks (iterate from previous incarnations) This should be a four paragraph section. The intro paragraph summarizes the three tasks. Each subsequent paragraph should start with a style Heading 2 with the name of the task and a more lengthy description. The descriptions should include screenshots of your actual interactive prototype in the style of either: a sequential storyboard, a state transition diagram, or a branching storyboard (see this Greenberg reading).

If the tasks changed from the previous assignment, please describe these changes and why they were made (e.g., was it based on your own design reflections, was it based on the experiences of TA06, etc.). You can also reference your TA09 Final Video here (this is called a video figure rather than a pictorial figure and is becoming increasingly common). So, if you have a skit in your video that goes over each of the tasks, you can refer to the video timeline in your prose (e.g., See 01:20-01:44 for a video describing Task #2). This should not replace your prose but should augment it.

4. Usability Tests

This section should have four sub-sections focused on the following. Each sub-section should be a Heading 2.
  • Describe how you pilot tested the interactive prototypes amongst the group and the resulting changes that were made either to the study protocol or to the final interactive prototypes themselves.

  • Describe how you recruited participants and their demographics (relevant to the project).

  • Describe your study method, which includes how you performed your study including the study protocol, the location, the length of the study, and a description of the data collection instruments (e.g., the video recording setup, the post-study survey).

  • Describe your analytical method, which describes how you analyzed the data you collected.

5. Results from User Testing

This section should detail the primary results from user testing and implications for design.

  • Provide a table of the quantitative results of your testing.
  • Summarize your qualitative observations, with a high-medium-low severity level for each one (for positive results, just note "good" or "positive" as the severity level).
  • Discuss the quality of your overall interface design. Identify limitations in the design (some of these may be due to the initial scope of the project). Identify design elements that work particularly well.
  • Provide recommendations for the next logical steps. Do you feel your design has been validated? Are minor adjustments needed, but the overall approach is validated? Is the concept still worth investigating but serious problems have been identified? Based on this conclusion, what are the logical next steps that you would recommend be done on this project?

6. If We Had To Do It All Over Again...

Reflect upon the design process you followed in this project. What aspects of the human-centered, iterative design process worked well? What aspects did not work so well? It may be effective to do this as a brainstorm session with your group. For example, you could start with questions like:
  • What were the most significant ways in which the design concept and the actual interface design changed under the influence of user involvement? What were the biggest surprises for you—the things you learned from or about users that you would not have predicted based on your own experience and intuition?
  • Did the methods you chose for your evaluation and prototyping get at what you were looking for? In hindsight, would a different approach (process, not specifics of your interface) have been better?
  • What were the most and least valuable among the methods you used, either generally or specifically for your project?

Appendix
The appendix should include the following. Each of the following bullets should start on their own page and should be titled Appendix A: <title>, Appendix B: <title>, etc. in Heading 1 style.
  • Your study plan
  • Raw notes from the three user testing sessions (notes should be clearly marked with a timestamp and session number)
  • A scan of the post-study paper survey responses (these should also have session numbers)
  • Scanned, signed informed consent forms