nvlabs / fpsci

Aim Training Experiments

FirstPersonScience Application

The FirstPersonScience application (FPSci for short) is a tool for conducting user studies on first person shooter (FPS) style tasks.

Standalone Release

You can get a stand alone version of FPSci compiled for Windows 10 here. In order to use this build, you will also need to install the x64 version (and possibly the x86 version) of the Microsoft Visual C++ Redistributable found at this link.

G3D dependence

FPSci depends on the G3D innovation engine which is a prerequisite for building FPSci. Installation instructions.

If you install G3D using its installg3d.cmd installer, then this will automatically install Python, Visual Studio Community Edition, and Subversion for you.

If you manually check out G3D, you'll need some data files from the research and game subdirectories so you can't just checkout the common subdirectory. The current version of FPSci requires at least G3D revision r7217 for the visual studio version upgrade, but most of the code will work withe earlier versions of G3D as well.

Build instructions

The solution file (FirstPersonScience.sln) can build and run using Visual Studio 2022.

Instructions for configuring and running an experiment

FirstPersonScience implements a simple mouse-controlled view model with a variety of parameters controllable through various .Any files (more on this below). The scene, weapon, target size/behavior, and frame rate/latency controls are all available via this interface.

When running, the keypad - button can be pressed to exit the application, or you can press TAB or ESC to toggle the user configuration menu, which has a "quit" button in it.

Organization and Terminology

Since FirstPersonScience was developed for use in scientific studies it adopts somewhat standard experimental terminology from psychophysics for much of its organization. This section describes the various levels of control in the application and provides the terminology by which they will be referred to for the remainder of this documentation.

The experiment is the highest level "grouping" of tasks in the application. An experiment is defined by a set of parameters specified in an experimentconfig.Any which include information about included sessions, as well as more general scene, weapon, and timing configurations.

A session is the next level of task grouping in the application. A session can reconfigure many of the (global default) properties setup in the experiment (it is assumed some property of interest is typically varied between sessions) as well as list of trials to include in this session along with their counts.

A trial is the lowest level grouping of task in the application. From a user-perspective the trial represents one or more targets being spawned, traveling, and either being shot or missed within a given time period. In specifying trials the developer enters trial groups, which are sets of trials specified using a target type id and count for one or multiple target types.

For more general information related to FirstPersonScience check out the documentation directory.

Config .Any Files

As mentioned above a variety of .Any files (similar to JSON file format) are used to control the behavior of the application. These files are as follows:

• startupconfig.Any is a simple configuration file for setting the playMode (debug feature) and pointing to the experiment and user configs
• experimentconfig.Any is the primary configuration for the experiment. It includes scene, target, and weapon information and also establishes the sessions to take place during the experiment
• userconfig.Any holds user information for (multiple) users including mouse sensitivity and DPI
• userstatus.Any keeps track of both the session ordering and the completed sessions for any given user
• weaponconfig.Any can be optionally included (using the #include("filename") option in the .Any format) to allow quick swap of weapons across multiple configs
• systemconfig.Any optionally configures an attached hardware click-to-photon monitor and provides (as output) specs from the system the application is being run on

All configuration files referenced above can be found within the data-files directory.

Managing Configurations

One reason for using the modular .Any file configuration structure shown above is the ability to track multiple experiments/sessions/trials without a need to perform major edits on the files. For example, an experimentconfig.Any file can either be renamed or can make use of the #include("[filename]") syntax provided by G3D's .Any parsing to include entire .Any files as sections in a larger file.

This allows users to create .Any files for a wide variety of experiments, weapons, users, and even system configurations, then swap between these at runtime by simply changing filenames or #include statements.

Repository Organization

A quick guide is provided below to this repository structure:

• source holds the FirstPersonScience c++ source files
• data-files holds scene and model files as well as the .Any files to configure the experiment
• docs contains documentation to aid experiment designers
• scripts contains some useful scripts for developers including some basic data analysis
• results is created when executing FirstPersonScience and holds the results .db files from each experiment

Source Organization

The source files (located in the source directory) implement a fairly typical G3D application structure with additional classes/files for many peripheral features. We outline the purpose of each of these source files below.

Application Code

All graphics and scene-related routines are in App.cpp and its affiliated header file. The App class inherits GApp from G3D and thus controls all rendering. Unlike many other applications this App class overrides the oneFrame() method from within GApp to allow us to reorder the render loop to reduce latency.

Several helper classes/files have been created to assist with application-level tasks.

• The PhysicsScene class/file is borrowed (w/ some modifications) from the G3D simpleGame example. This class extends the typical Scene by adding support for a tree based collision model, using a sphere as a proxy for the player.
• PlayerEntity class/file is also borrowed from the G3D simpleGame example. The class provides a controllable player-entity (rather than no-clip camera) that works well with the PhysicsScene
• TargetEntity class/file is a custom class implementing different target types that inherit a common, generic TargetEntity (which in turn inherits G3D's VisibleEntity)
• The GuiElements class/file contains some custom GUI menus designed to work specifically with this application
• The Dialogs.h file contains some generic dialog box classes that are used for asking questions of the user following a session.
• The PyLogger.h file encapsulates the calls used to launch the LDAT-R python logging tools from within the application when running FPSci with an LDAT-R device present.

Experiment/Session Code

The majority of experiment/session level management occurs within Session.cpp and it's affiliated header file. This includes configuring sessions, controlling the per-trial/session state, and logging results.

Experiment/session-level helpers include:

• Logger.cpp / Logger.h which manage logging for the session
• ExperimentConfig.h which contains the .Any serializable classes for all session/experiment control
• Param.h which is used as a sort of dynamically typed dictionary to help consolidate parameter passing

fpsci's Issues

Unnecessary setters

There are several set routines in App.cpp that are only called from within this file to set private member fields. This could be accomplished simply by setting the variables instead of calling setters. Should these setters be removed? These include:

• setSceneBrightness()
• setDisplayLatencyFrames()

Initial Click to Spawn Target

When you first open the application (playMode does not matter) and close the tab menu no target is spawned. In order to spawn the first red target you have to left click once. It seems as if this target should just appear without clicking, but I want to verify this before "fixing" the issue.

Latency box reduced size

When the latency box is enabled, it currently consumes an entire vertical section of the screen. We should reduce the area consumed by this, and reduce the box size to work for the latency monitor placement we care about.

If someone is extra interested, we could add a control for latency box placement to the experiment config.

More UI support for motion recording mode

• A hot key to start/stop motion recording (we can record motion without opening user settings window, and also in full screen mode).
• Some indication that recording is happening (useful in full-screen mode). Maybe a line of sentence at the top.

Target (or enemy) behavior: attack

From near-term to long-term:

• Enemy shoots at the player at pre-defined frequency and accuracy (can be random variables).
• Enemy weapon can be configured as hitscan or projectile.

Below are ideas that are not critical.

• Enemy aiming strategy may be implemented as a dynamics model.
• Enemy aiming strategy may be implemented as an ML model.

unlockFramerate

The unlockFramerate control flag in App.cpp currently just sets a very high effective frame rate (8192fps). We could either document this for users to do through the existing refreshRate parameter in the session configuration of experimentconfig.Any (remove this flag) or we can document it/include it as an additional option for the per session configuration in the experimentconfig.Any file.

Font and HUD Settings

We allow these to be turned on when playMode = False, but right now they can't be configured in experimentconfig.Any (see issue #14).

Should we add controls for this to the experiment config? If so how should they be formatted?

Walking Player Motion

According to @morgan3d it should be fairly easy to add a simplified "physics" based walking mode to the game based on the simpleGame example in G3D. I have started looking into this as a configurable option for the application.

App.h Private Members

There are still a number of public variables in App.h that should either be removed, protected, or moved into Experiment.h. These include:

• m_targetDistance
• m_spawnDistance
• m_projectileSpeed
• m_projectileShotPeriod
• m_projectileSize
• m_presentationState
• m_reticleColor
• m_isTrackingOn
• m_t_lastAnimationUpdate
• m_t_stateStart
• m_t_lastProjectileShot

There may be more, feel free to add these to the discussion.

Crouch in walkMode

When walkMode=True there is no support for crouch currently. This can be bound to the Ctrl key as is typical. We should implement a crouchHeight parameter similar to the playerHeight parameter to set the crouched height.

Storing gamer/camera pose

Since the gamer is allowed to move we have to store the 3D pose in the player action table. I have implemented a version and it's working since I don't have access I can do pull request if it's fine with everyone. Btw, I suggest keeping the current azimuth and elevation and just add position + orientation columns. This will allow a post-hoc analysis & sanity check for 3D calculations.

Decals Casting Shadows

It seems that with the right lighting our bullet hole decals can cast shadows as seen below:

This should be corrected so bullet holes do not cast shadows in the scene.

World-space Parametric Target Motion

We would like to develop a world-space, parametric target motion that is easily configurable for various levels of challenge.

The goal is a simple randomized motion within a predefined "pen" or "box" specified in the target's config. This would reuse the existing destSpace field to allow non-destination based (flying and jumping) targets to use the destSpace=world value as well. In this case an additional bbox field will be required in order to create an area in which to randomize target motion.

Exact parameters of this target motion are TBD, but we can start w/ a random point-to-point flying motion. Eventually we might add support for other (less linear) motion paths as well.

This approach could be accomplished by a script that produced a target.Any file with random position inside a bounding box and relied on the destination-driven positioning mechanism.

Eventually a "smart target" would include collision detection/avoidance and additional understanding of in-game mechanics that would require better understanding of the scene than could be provided by an external script.

Floating Combat Text

We have discussed adding floating combat text to our application as another option for indicating hit/miss or damage to the target as many games do. This could be a good option, and comes with several parameters we probably need to add to config files.

We never implemented this but it should be comparable to adding a floating health bar as discussed in issue #23.

Target motion/dodging in onAI

Target motion/dodging (and attack) to be implemented (or factorized if already exists in onSimulation) in onAI.

Player Spawn Heading

Currently the player rotation stored in the scene.Any file is overwritten by the m_heading field in the PlayerEntity. This field is always initialized to 0 so the initial player heading is effectively "fixed". We should add an Any serialize-able field to allow user to specify the player's initial heading/view direction.

resetView()

The resetView() method on line 1257 of App.cpp is not used and can be removed. Should we do this?

Multiple targets

It seemed easy at first, but soon I realized it requires changes across many parts. We should the decisions on them based on a high level philosophy, and then implement it. As a starting point of discussion here are the local changes needed.

• Health point tracking.
  • Currently m_targetHealth is one member variable in the App class. Probably we want to move away from it and have each entity track its own health.
• Hit detection mechanism in the App class.
  • Do we want to define multiple FlyingEntity's or JumpingEntity's and destroy each of it when health becomes 0? Or we keep all the entities but set the health values to 0 when cleared, but destroy them altogether at the end of a trial? In the latter case, we would need to check health value after checking intersection of the ray.
• Log writing method in the Logger class.
  • Arbitrary number of objects can be in one trial, and objects can get destroyed in an arbitrary order. How can we capture individual objects' trajectory in the log? Generate tables as many as the maximum number of objects? Or multiple columns in one table? Or multiple values in one column in one table (e.g. comma-separated values inside one column)? How should we handle objects cleared before a trial ended?
• Configuration format in .Any files.
  • What's the optimal way of doing it? My first idea is to define motion types as we do now, and use a nested dictionary format inside the "id" field in "sessions" / "trials".
• Configuration loading methods in various Config classes.
  • This depends on the new configuration format.

testCustomProjection

Currently, the testCustomProjection flag in App.cpp is used to conditionally apply a fixed pixel-shader distort.pix when set to True. However, we could allow arbitrary shaders (w/ the same inputs) to be run by adding a shader text field to experimentconfig.Any. If this field is empty no shader would be run.

We could also just move testCustomProjection (or some version thereof) into the experimentconfig.Any file. Alternatively, since currently testCustomProjection is hard-coded to False we could just remove this altogether (and drop the shader directory from data-files).

Allowing Player Motion

Allowing in-app player motion is simple enough to add to the experimentconfig.Any file, but may have other impacts on the application. Namely we should look into how this impacts spawning/moving targets. This could likely be tested in playMode = False.

Cylinder target

Add a new target shape, a cylinder. This represents the target shape in FPS games more realistically (standing figures).
Note that we want a black boundary as in our current sphere target.

Walk Mode

There are a number of issues that still occur/come up when walkMode = True. Some of these are listed below:

• The PlayerEntity definition in the scene .Any file needs to be removed when walkMode=False and added when walkMode=True, try to unify these to use a single scene
• Gravity/jump velocity seem to be correct, but the "zero" velocity seems very short (feels like hitting a ceiling)
• Depending on gravity settings climbing stairs can be possible/impossible without jumping, and there can sometimes be a tendency to "fall down" stairs without moving
• When setting a 0 crouchHeight in the experimentconfig.Any file the player still seems to be a bit above the ground

Target Size should mean something

In the config files when we set a target's visualSize value, the size controls the scaling relative to other size values, but it doesn't have a meaning in world space or visual angle or anything related.

We should adjust the scaling factors to make the target size set the world space size of the target. I recommend that the size variable should set the horizontal width of the target in meters. So a value of 0.10 would mean 10 centimeters.

This will interact with the distance setting and probably will mean breaking compatibility with older experiment configs since realistic target sizes will make the most sense further away from the viewer than we've been placing them so far.

Hit/Destroy Player Action

As we implemented multiple targets towards issue #28, we broke the accumulation of the hit and destroy player actions (these seem to be recorded as non-task?). We should look into this and get it working again.

Commented out spawnTarget() call

At line 93 of App.cpp there are two commented out calls to spawnTarget().

//spawnTarget(Point3(37.6184f, -0.54509f, -2.12245f), 1.0f);
//spawnTarget(Point3(39.7f, -2.3f, 2.4f), 1.0f);

Since this isn't how targets are spawned in the actual application we might want to remove them. This is affiliated with issue #11.

Target Health Bar

There is commented our source for a target health bar (drawn in a fixed position in screen space) at line 847 of App.cpp do we want to leave this here as a reference or should we remove it since the target changes color? Do we want an option in the experiment config file to control this?

Key Frame Editing in App

Based on the work described in Issue #44 we've implemented a simple key frame path creator tool when playMode=False in startupConfig.Any. Currently, the user can move about the map and either:

1. Toggle the menu to use GUI buttons to drop time-spaced waypoints around the map.
2. Use the q key can also be used to quick-drop waypoints at the interval set in the menu

In addition the last waypoint can be removed, or the entire set of waypoints can be cleared. This is an issue to track the remaining implementation work and other suggestions for features.

Additional Items to Implement

This is a starting point for a more elaborate path built-in target path editing system. Additional features to support include:

• Record realtime/time-scaled player motion
• Load existing target waypoints to edit/create new paths from
• Preview a designed path using a requested target model
• Alter the position of individual waypoints on a path
• Visualize the time between waypoints
• Edit the list of waypoints along with their timing in a GUI element
• Create bounding boxes for random target motion

Stairstep Experiment Specification

The SingleThresholdMeasurement.cpp file has code in it to perform stairstep-style stimulus. This could be leveraged in abstract-fps. Currently we're not sure how this behaves (as-is) and don't have a model for how the user specifies the parameters its should control/how they impact task difficulty. Needs more discussion before any implementation takes place.

Eye tracker calibration procedure

How the calibration procedure for eye trackers should be treated? Two obvious options.

1. Mode. We have calibration mode and experiment mode. In calibration mode we see 2D graphics and user interaction is limited to continue or stop the calibration procedure. In experiment mode we keep all the interactions we used in targeting experiment (view direction change, UI pop up, etc).
2. Experiment. Calibration becomes another experiment in parallel with those such as 's1-training' or 's5-real'.

Bullet Hole Decal Clipping

Bullet hole decals occasionally clip through walls. Not sure what is going wrong, but looks like something affiliated with the normal from the ray cast not quite lining up with the wall.

Player object for hit detection on player

We need an object that represents the player for hit detection. This should be invisible in first person view. Maybe a transparent sphere or cylinder to begin with?

Multi-jump

As part of an effort to port all motion to what has up to now been referred to as walkMode=True we realized that jumps can be "infinitely stacked" in this mode. We need to put in place some sort of mechanism to limit the player's ability to jump as high/as many times as they'd like.

This is related to #36.

variableRefreshRate Control

The variableRefreshRate control flag is misleadingly named, and is also hard-coded into App.cpp. We should decide whether to rename/add to an Any config file or remove.

Bullet source

When the renderBullets flag is set to True in the weapon configuration within experimentconfig.Any the bullets only align with the barrel as a special case for the current weapon model, we should include configuration for this offset. Additionally, it seems that when the bullet collides with the target its initial position seems to change relative to the weapon. General odd behavior has been observed when renderBullets = True.

Initial Motion Allowed

When the application is first run, the player can move (regardless of the playMode flag's values) until they close the user GUI with tab. Once the GUI is closed the player can no longer move. This is related to the m_userSettingsMode flag's behavior in App.cpp.

This issue is related to issue #15.

Key Frame Target Motion

I believe that issues like #42 and #37 require us to first implement a more robust target motion type. Ideally this is one in which the target is located "in the level" rather than orbiting a small sphere around the player's view. This would also allow targets to be sized using world-space dimensions rather than arbitrary scale factors. This is a place to capture discussion related to this implementation as it moves forward.

Audio should work in developer mode

Audio is currently coupled to the playMode/developerMode setting. It would be more convenient to be able to enable or disable audio independently from the developer mode. This will enable GUI menu control to mute audio.

At some point we probably also want to add volume controls as well.

Spawning Targets

There are multiple mechanisms by which to spawn targets (primarily in App.cpp in the application right now. It is not entirely clear to a new user which are actually being used. There are also several TODO comments indicating we were planning to move towards @morgan3d's approach to target motion but didn't have time before our first study. Do we want to do this now? Should we remove the unused methods? Should we unite how targets are spawned into a single routine?

Hit detection be consistent with weapon dynamics

Hit detection on player and target needs to be consistent with weapon type (projectile must be evaluated when it intersects with player/target).

Field of view settings

The field of view is currently set with horizontal field of view. Some people may want to set it vertically. Someone should add this option to the experiment config in a sensible way.

Note that we don't want destructive interference when the experiment config attempts to set both. It's almost certainly never going to be intentional to have the two settings fight.

playMode = False

The behavior of the application when playMode is set to False in startupconfig.Any is rather different than that when playMode is True.

Differences to note include:

1. The way in which targets are spawned (see issue #11)
2. The ability to select an aiming reticle (but not save it back to any config file)
3. The ability to overlay HUD not present in the experiment
4. The ability to move about the map
5. The inability to access/preview any sessions from the experiment config

There may be additional differences not accounted for here. This issue can be referenced in creating future issues related to differences in behavior.

Target Respawn Count

For some experiments it would be useful to allow targets to respawn after being destroyed (within a single trial). In this case the TargetConfig could include a respawnCount that is set to0 for a single spawn, -1 for infinite spawns, or any other finite number for a fixed number of spawns.

The waypoint tracking system will also need a time offset added to allow respawned targets to start their paths over as opposed to continuing from where they were destroyed. This could be made an option.

Weapon cooldown indicator improvement

The current weapon cooldown indicator is placed on the side of the screen and takes up a large portion of the field of view. Let's discuss and implement a change that more closely matches what is done in games.

Proposals I can think of are the following:

1. A circle around the aiming reticle that fills clockwise after each shot.
2. A weapon model that recoils

Are there other weapon cooldowns worth considering? I think 1) above may be the easiest since it's at a fixed location in screen space.

Player health bar

Player health state as health bar.
A large, eccentric red ring when player health decreases (receiving hit from enemy).

Audio volume control

We should add volume control for the audio as alluded to by #52

Setting Materials for OBJ

The amPose->materialTable.set() calls in App.cpp uses an explicit OBJ name (core/icosahedron_default). This is a problem if we want to allow the new target defined modelSpec to use OBJs without this internal name. For now our new cylinder model just uses this name internally. In the future it would be good to get this OBJ name rather than hard-coding it if possible.

spawnParameterizedRandomTarget()

The spawnParameterizedRandomTarget() method is implemented but not called in App.cpp. Should we be moving to use of this method or removing it? This is related to issue #11.

Targets Table

As part of implementing issue #28 we created an issue in what was formerly the Conditions table (now the Targets table). Though all target setups are written to log.txt as SQL INSERT statements, a smaller number appear in the final log. Need to figure out why these aren't being added correctly.

Also we haven't yet figured out a way to get the target's ID into the Targets table. This will be a code structure challenge, since the table is written before the target entities are spawned.

Rotational Keyboard Controls

The Q and E controls for view rotation (left/right) are connected to rotation, they should probably be turned off.

Empty onCleanup()

The onCleanup() method in App.cpp is empty/still included with just comments in the body. Should we leave this around or remove it?