TEST: testscripts & questionnaires (lesson 6)

For questionnaires go here

Testscript

 The research is performed on DD/MM/YY in the
 VR/AR atelier of the Faculty of Digital Media and 
 Creative Industries, by ....... (name facilitator). 
 Our test is performed by ....(number) respondents. 
 Profiles of the respondents can be found here (link)

Purpose & Goals Testing

Take some time to define your goals and the purpose of your testing. Set up a good plan for testing.

Main research questions (examples) tip: look at the menu on the lefthand side. Use the topics of this course for defining your research questions!

1. Storytelling & Presence: - how do the respondents explain the story - how to the respondents explain the virtual place where they are in?

2. Interaction Techniques: Wayfinding - how do respondents navigate through the space, how do they learn where they are? - how do they get out? (cooperate, sign language, cues)

3. Interaction Techniques: System control - how do respondents navigate through all the puzzles, how do they learn about cues? - how do respondents experience the cues?

4. Social VR: Virtual Identity - how do the respondents explain their multiplayer personality in this escape room (powerless, fierceful, strong, awkward, sloppy, massive, powerful, deteriorated, hurt, deceitful) - how do the respondents explain their personality in this escape room (powerless, fierceful, strong, awkward, sloppy, massive, powerful, deteriorated, hurt, deceitful)

5. Social VR: Cooperation & Communication - how do respondents communicate? 
- how do respondents work together?

Administrative step

• Consent form: fill in and sign (example & example for children)
• Information sheet with basic characteristics of the participant (e.q. visual abilities, height, hand measurements)

• Questionnaire for background information, for instance their VR/AR experience and gaming abilities (hours/week)
• Interview background information participant
: for instance sensitivity, fears or any other relevant topic.

Interview background information

• Can you explain shortly who you are and what you do? Topic number 1: Escape Room

• Some questions related to the topic. ‘do you know the concept of an escape room? ‘ have you ever been in a physical escape room?’ ‘ can you describe this escape room?’ ‘did you do this with others, how did you work together?’ ‘ what was most exciting?’ ‘what was not exciting?’ Topic number 2: Communication

• Some questions related to the topic. ‘ Have you ever done conference calling or do you use FaceTime?’ ‘Do you use it for work?’

Preparation

Procedure

Since this is a Walk-Through test, respondents perform tasks as autonomously as possible and we try to adjust the scenarios to their situation. This test is not focused on very specific tasks, but you can create tests that revolve around very specific tasks and then measure the quality of the interactions, performance, comfort, time completion etcetera.

Scenarios

You can set user journeys / scenarios, different routes that the user can take and then later interview them about these specific routes.

Tasks

You can list tasks here. These can be specific or a set of tasks that are related or need to be done in a specific order to accomplish the task.

Evaluation Measures (what you can measure)

https://www.coursera.org/lecture/3d-interaction-design-virtual-reality/evaluation-metrics-Itlsd

Think about possible evaluation measures that can be collected and calculated during the test, both quantitative and qualitative evaluation measures. Below you’ll see some of them:

For the quantitative evaluation you can measure:

• The numbers of collisions during single tasks, (Collision Avoidance test)

• Time to complete each task, (Time Completion test).

• Errors made (while estimating the relative distance (Access Width Estimation test)).

• Number and percentage of tasks completed correctly (Collision Avoidance and Access Width Estimation tests).

For the qualitative evaluation you can measure (questionnaires):

• Adequacy of the task to the application, (Adequacy to application).

• The realism of the visual feedback, (Realism).

• Sense of Presence, (Presence questionnaire) .

• Depth impression, (3D impression). (“how big was the space in VR for the user?”)

• The user’s viewing comfort, (Viewing comfort).

Evaluation Metrics (extended)

Performance Metrics

Timing accuracy

Timing accuracy is critical as it is highly related to a variety of cybersickness, such as nausea, disorientation, and headaches. We define the following two metrics to quantify timing accuracy:

• Initial delay: The time difference between the start of head motion (t0) and that of the corresponding feedback in the virtual world (t1).

• Settling delay: The time difference between the stop of head motion (t2) and that of the corresponding feedback in the virtual world (t3).

• Positioning accuracy: The spatial inconsistency between physical moves and visual feedback is another major cause of cybersickness. We accordingly define two metrics as follows.

• Precision: The angular positioning consistency between physical moves and visual feedbacks in the virtual world in terms of degrees.

  Sensitivity: The capability of HMD inertial sensors to perceive subtle motions and subsequently provide feedbacks to users.

• Performance Measurements of Virtual Reality Systems: Quantifying the Timing and Positioning Accuracy

Task Performance Metrics

• User-centric Performance Metrics
• The effects of substitute multisensory feedback on task performance and the sense of presence in a virtual reality environment

Subjective Response Metrics

A subjective response refers to the personal perception and experience of the interface by the user. (e.g. perceived ease of use, ease of learning, satisfaction etc.) These response are often measure via questionnaire or interviews. Some of the topics that are tested:

• presence

• user comfort

• cybersickness

• simulator sickness

Execution

Introduction (c.a. 5 minutes)

Before we start the test, we will explain to the participant:

• The participants are not the subject of this test. Nothing they do can be considered right or wrong. We test the idea of a product and want to know what does work and what doesn’t work.


• Please try to think out loud while performing the test. You can say anything you’d like. There is no right or wrong. We really would like to know what you think of the idea or product.


Facilitator tasks

While the respondents perform their tasks, the facilitator makes sure that the respondent acts autonomously and she/he intervenes when the respondent gets stuck. Besides this, the facilitator and observant pay attention to:

• Expectations: do all functionalities work as expected?

• Understanding: are all functionalities and steps clear and understandable?

• Behaviour: how do they look? what do they do? what objects and elements are being interacted with?

• Experience: how does the respondent appreciate and experience the idea or product?

Evaluation questions (for interviewing)

In this part you will ask all sorts of questions that will help you to understand how the respondents have experienced your prototype.

Examples:

• What did you just see?

• Can you explain what you thought of this? What did you like? What didn’t you like? What can be improved?

The questions below need to be translated to the interview setting, so don’t ask these questions directly:

1. How much was the respondent able to control events?

2. How responsive was the environment to actions that the respondent initiated (or performed)?

3. How natural did the respondent’s interactions with the environment seem?

4. How completely were all of the respondent’s senses engaged?

5. How much did the visual aspects of the environment involve the respondent(s)?

6. How much did the auditory aspects of the environment involve the respondent(s)?

7. How aware were the respondent(s) of events occurring in the real world around the respondent(s)?

8. How much did your experiences in the virtual environment seem consistent with the respondents real-world experiences?

9. Were the respondent(s) able to anticipate what would happen next in response to the actions that the respondent(s) performed?

10. How well could the respondent(s) identify sounds?

11. How well could the respondent(s) localize sounds?

12. How compelling was the respondent(s) sense of moving around inside the virtual environment?

13. To what degree did the respondent(s) feel confused or disoriented at the beginning or breaks or at the end of the experimental session?

14. How involved were the respondent(s) in the virtual environment experience

15. How distracting was the control mechanism?

Source PQ Presence Questionnaire: https://nil.cs.uno.edu/publications/papers/witmer1998measuring.pdf

(Optional): other scenarios not in the prototype yet

If your prototype does not contain all aspects that you would like to test (which is regularly the case in your project), then you can still try to receive information from the participants about your ideas. Of course, it’s not fully accurate, but it can help you in further development of your product. For instance, you can go back to a certain part in your experience and then ask your participant about their experience, and then move further to the scenario. Or: You can show storyboards as well and then let the participant help you to fill in the gaps.

“Suppose you are………..and you see………. You are curious and you figure out that you need to put on your VR-headset to understand what you need to do.”

Example from lesson 6

https://vimeo.com/329848889vimeo.com