mayacakmak / se2 Goto Github PK

We can move discussion about data to this issue.

One initial TODO:

• Write a script that converts (probably a subset of) the data into a CSV (spreadsheet compatible) format so we can more easily browse, take notes on, code, etc. open ended answers

I have looked into both how Google recommends storing data in Firebase, and what kinds of events and data the programming is currently generating. Google recommends mainly to keep the structure as flat as possible, and to denormalize it when needed.

This structure is generally what I came up with. It has 4 main datatypes:

• Users: The anonymous users
• Interfaces: The different test interfaces
• Cycles: A set of actions from randomly generating a target location to the user correctly moving the end effector there
• Actions: A single action or state change that takes place as part of a cycle

{
  // Users contains only meta info about each user such as session start time
  //  and any other information we need to save about them
  // This is stored under each user's unique ID
  "users": {
    "u1": {
      "date": "Wed Jun 10 2020",
      "time": "11:45:13 GMT-0700 (Pacific Daylight Time)",
      "timestamp": 1591814713656
    },
    "u2": { ... },
    "u3": { ... }
  },

  // Each interface type is split up into users by user ID
  // which each contain a list of cycle IDs
  "interfaces": {
    "arrow-press/release": {
      "u1": {
        "c1": {
          "totalCycleTime": 2, // Seconds
          "numberOfClicks": 4
        },
        "c2": { ... },
        "c3": { ... },
      },
      "u1": { ... },
      "u2": { ... }
    },
    "drag-click": { ... },
    "panel-press/release": { ... }
  },

  // Cycles and actions are separated from users
  // so we can still query individual events easily
  "cycles": {
    "c1": {
      "a1": {
        "date": "Thu Jun 27 2019",
        "eventName": "ring-release",
        "newState": "cursor-free",
        "prevState": "rotating",
        "time": "16:40:27 GMT-0700 (Pacific Daylight Time)",
        "timeStamp": 1561678827981
      },
      "a2": { ... },
      "a3": { ... }
    },
    "c2": { ... },
    "c3": { ... }
  }
}

User, Action, and Cycle ID's would probably randomly generated by Firebase, I just used easier to read strings here.

This structure should allow us to fairly easily and efficiently query cycles by either interface or by user. We might want to edit the structure if we will need to query the database differently. I would love any feedback you have on this. If it sounds good to you, then I can start implementing it!

Here is the completed video for one of the interfaces:
https://youtu.be/-bwEiM_2Ur8

Any feedback/sticking point/missing things before I move onto making the other 7 videos? @kaviMD

Once videos are finalized we need to update instructions.html to (1) get which interface the participant is using from the URL, (2) display the correct video for each interface, and (3) figure out the pipeline from MTurk to the full set of interfaces (probably different HITs for each interface, then each HIT has a different link with the URL parameters).

Since our meeting, I have implemented two more ways to display and interact with a URDF in a browser.

The first, is pure js, it uses a three.js (a 3d rendering library for javascript) and a URDF loader. This is the code: https://github.com/mayacakmak/se2/tree/urdf-visualization-testing/urdf-visualization/only-js

The second is a bit more complicated. When the website is initialized, it opens up anywhere a ton of ROS instances, each with their own visualizer, simulation, etc. Then, whenever a user visits the website, it assigns them a specific instance which they can interact with. The website also detects when they leave and opens up that instance for another user to interact with. The number of users we can support is limited by how many ros instances the server is capable of running at once, but it does open the door to having one gazebo simulation per user or something like that. This is the code: https://github.com/mayacakmak/se2/tree/urdf-visualization-testing/urdf-visualization/multiple-ros-instances. The backend is written in Flask with socket-io for communication and detecting when a user connects or disconnects.

Basic instruction, link to study, place to enter completion code.

Currently the interface has 'exact' targets, where the SE2 object (dot and line specified by an x,y position and theta rotation) needs to be taken to an exact target pose (even though there is a small threshold around the exact pose). We would like to be able to specify flexible targets, where the goal is to bring the object within some range near a given SE2 pose. We can do this by changing that small threshold but we need the visualization of the target to reflect the thresholds.

• Flexible position: This can simply be a larger circle at the center. We need to ensure: (1) the threshold is applied to Euclidian distance, not x and y separately (L1 distance); (2) the desired rotation is still clear (e.g. dashed line over the larger circle, perhaps extend length of the rotation line by radius of circle).
• Flexible rotation: Use an arc/pizza slice-like visualization to represent flexibility in rotation. Ensure the position (small point) is still distinct.
• Both (optional): Not sure this is feasible or necessary, but perhaps good to have the option.

I see progress on the 3d-interface branch but haven't been able to test it. @kaviMD what is the status? What remains to be done? How could I help?

First version completed by Kavi, some formatting edits by Maya.

Revising list of questions in the Google doc.

• Finalize questions
• Finalize question formatting throughout
• Add code to send the form data to the database to attach to the current users data.
• Red asteriks on required questions
• Is the data remembered if the user goes back and forth?

The practice screen randomly specifies the target after each trial. For the systematic data collection we need to systematically vary:

• How far the target is to the starting point (center) -- randomize which quadrant it is in to balance other potential biases
• How large the target area is (once we have flexible x, y -- issue #6)
• How large the target rotation slice is (once we have flexible theta -- issue #6)

We need to keep the total duration of the experiment within a certain range to avoid fatigue, so we can decide how many variations of each of these we want to do based on how long a single session tends to take for the slowest interface.

Sampling variations: Rather than having fixed/exact variations we might want to have a complete set of exact options and add a little bit of noise to everything (center_x, center_y, center_theta, thresh_xy, thresh_theta => note that the variations are in terms of dist_xy, dist_theta, thresh_xy, thresh_theta, but noise in the former would result in less recognizable patterns).

The panel interface was implemented to support clicks (single small move of the object) as well as press-hold-release (for longer continuous moves). The latter is used much more than the former. A click-based alternative to that is that when the arrow is clicked once, the object starts moving; when it is clicked again, it stops moving. Should be reasonable to implement following the template of the other interfaces, but will require some re-discovery.

Codepen had a nice automatic compilation of .pug files into .html. Perhaps sure GitHub has some plugin that helps do that? If either we might want to create some sort of compilation script (possibly with instructions to install pug on computer) or we can also switch to editing HTML directly and remove the pug files.

Look into security rules on Firebase

@kaviMD Creating a separate thread for study 2 related data stuff. The first MTurk batch is not running so soon we should have some data to look at.

I know you've updated the data model a bit in on Firebase. Is the data processing code under se3/ also updated? What needs to be done there?

Data we'll want to extract from the logs:

• Task completion time (we might actually want to look at these per task since the task order is kept constant and some tasks might highlight differences more)
• Mouse clicks
• Time spent in each view (not sure there will be differences across the interfaces, we might aggregate, but still useful to understand how se2->se3 worked out)
• Questionnaire responses

Other ideas?

Todos:

• Log low-level actions to a different section of the database (Only for SE3)
• Update logging to work with SE3
• Add question ID's to questionnaire (Only for SE3)
• Look into odd questionnaire results from Study 1

While most controls do not allow the object to get outside the workspace, the panel control can send the object off limits--we need to implement check so the object doesn't move once it hits the walls of the workspace.

We probably need to tell people to use Chrome on a desktop. Could we automatically detect if people are on a mobile device and not let them participate? @kaviMD