18f / culper

Culper is the foundation for eApp, a part of the National Background Investigation System (NBIS), supporting the SF-86, SF-85, and SF-85P. This is its source code and developer documentation. For background information and a demo video see https://github.com/18F/culper/wiki

Project Culper

To create the applicant questionnaire for the National Background Investigation System (NBIS), the project team is employing a user-centered design approach leveraging key principles from the U.S. Digital Services Playbook:

1. Understand what people need
2. Address the whole experience, from start to finish
3. Make it simple and intuitive

Culper is a responsive, mobile-friendly web application that applicants can use to fill out an SF-86, SF-85, or SF-85P. It forms the foundation of eApp, which is part of a larger NBIS information technology system.

Table of contents

• Project Management
• Development
• Architectural diagram
• Additional
• Contributing

Project Management

The project team primarily utilizes a private NBIS tool to administer user stories, defects, and tasks. Users of this open source code however are welcome to report issues via GitHub. Higher-level and non-development tasks are tracked in a Trello board.

Documentation for this software is maintained under docs and in a private wiki.

Development

Initial setup

Dependencies

• git
• docker
• docker-compose
• make

For more information on licenses and third-party source code, use a tool like this one.

Clone all things

Clone the repository and cd into it:

git clone https://github.com/18F/culper
cd culper

Then to develop locally, create a .env file:

cp .env.example .env

For more information on the various settings, examples, and values please refer to the configuration documentation.

Tests

To do the initial setup and ensure that all tests pass locally:

make

Setting up the database for API integration tests

cd api
make reset-test-db

Pre-Commit Hooks

There are some pre commit hooks added using husky to run them. The pre-commit hooks are designed to run outside of docker for speed so you will need to have run yarn install outside the containers if you have not done so.

The hooks should automatically be applied once yarn install is complete, and will run the eslint and css lint rules on files changed that are staged for commit. Developers should apply lint fixes; however, specifiying --no-verify as an argument to git commit to bypass this check for temporary wip commits.

The eslint rules for the pre-commit hook will be using a stricter standard than the CI build to allow us to improve the code as we modify files while avoiding having to fix all the errors that would be generated if the rules were applied to the whole project immediately. Eventually we will want to apply the stricter rules to the CI build as well.

Running a local server

To run a local server, we are using docker containers leveraging the docker-compose tool via the command:

make run

Then direct your browser at http://localhost:8080. The access the site in development use the username test01 and password password01. If you make changes to frontend files, the site will automatically rebuild after ~10 seconds.

How it works

The Make target calls Docker Compose, which then runs containers for various parts of the system. Frontend assets are built from their own containers into the dist/ folder, which are then served by nginx. There is also an API backend (under api/) written in Go, which has a PostgreSQL database behind it. See the architecture diagram below.

See also: frontend docs.

Architectural diagram

There are several possible architectures which may be implemented. The diagram references one of those possible solutions and highlights the basic flow of data within the system. It also demonstrates integration with external systems (e.g. identity services) which are not part of this project but may be part of the overall system.

Additional

See advanced docs for more.

Tooling

See tools docs for more.

Formatting

Supported files are formatted using Prettier, though note this should only be done when a file is new or heavily modified. You should install Prettier for whatever editor you use.

Linters

• Vim users: install syntastic
• Emacs users: install flycheck

For command-line alternatives there are the following:

• For CSS, run make lint-css
• For JavaScript, run make lint-js
• For HTML, html-lint which may be installed with yarn add html-lint

Contributing

Please refer to the contributing documentation.

culper's Issues

Save button - animation and timer

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2915

• The saving animation is not working on the save button
• Looks like the timer is off, after a save it jumped back to 14 min ago

Scrolling quickly causes page to bounce/vibrate

From e-QIP-prototype-truetandem created by ryanhofdotgov : ryanhofdotgov/e-QIP-prototype-truetandem#2884

Under 0.5.1 (Sprint 35), latest Chrome, macOS. See attached quicktime video.

On one page, used the scroll wheel on my mouse to quickly get to bottom of page to click next. Once I got to the bottom, the browser would continuously show the bouncing/vibration effect depicted in the video. Scrolling up stopped it, scrolling down quickly again triggered it.

scrolling.mov.zip

Separate database migration process from application runtime

Backend api process will automatically execute SQL in files in the migrations directory, depending on the value of DB_MIGRATION_TARGET and the skip-migration command-line parameter.

Improve privilege separation by:

• moving migration code into separate tool
• reduce privileges of PostgreSQL account used by backend api executable so as not to have any DDL modification permissions.

Mitigated by container deployment environment (container access restrictions, file system permissions, etc.) where write access to migrations implies ability to affect the entire backend.

SSN on review screen

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2836

On review screen when tabbing through SSN inputs:

• Tabbing through inputs without changing any numbers required the user to re-confirm their SSN. Can we remove this?

It requires extra work for the user and may be confusing screen reader users

Test eQIP web service submissions and automate tests

From e-QIP-prototype-truetandem created by pjsumm : ryanhofdotgov/e-QIP-prototype-truetandem#2876

As a eApp developer I want to test submitting data in all fields of the eApp online application form to the eQIP test/integration site via web service so that I can validate that the data submits without error.

Acceptance Criteria

• Create test automation design
• 18f approval of test automation design
• Verify all form data submits without error. See #2850

Cookie used to send auth token to front-end after SAML sign-on scoped to wrong domain

After successful SAML sign-on, the backend sets a cookie in the redirect response, but the cookie is scoped to the fully-qualified domain name (FQDN) of the frontend. The backend has no authority to do that, and so the browser does not expose the cookie to the frontend on the subsequent request.

I believe that this wasn't caught during the initial development because testing was done using localhost as the FQDN for the frontend and backend.

WSO2 POST to `/auth/saml/callback` does not set Origin

After successful authentication at WSO2 and POST back to /auth/saml/callback on the backend, WSO2 seems to trigger a condition that causes the browser to set Origin to null:

POST /auth/saml/callback HTTP/1.1
Host: api:3000
Connection: keep-alive
Content-Length: 6998
Pragma: no-cache
Cache-Control: no-cache
Upgrade-Insecure-Requests: 1
Origin: null
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9

This affects the CORS setting and might be caused by the privacy contexts listed here:
https://wiki.mozilla.org/Security/Origin#Privacy-Sensitive_Contexts

Navigate between sections using keys (Vim and/or arrows)

From e-QIP-prototype-truetandem created by bmallred : ryanhofdotgov/e-QIP-prototype-truetandem#247

It would be nice to navigate between sections using keyboard shortcuts.

Section 1 - first screen back button

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2865

• Change text on first back button to read:
  "Back
  Information intro"

Currently is says "Identification"

Accessibility - Employment activities- tab order

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2849

"Additional periods of activity with this employer"
Yes/No

• Tab order is broken here, next tab after this question goes to "Skip to main content"

Update styles for Print PDF

From e-QIP-prototype-truetandem created by floresj : ryanhofdotgov/e-QIP-prototype-truetandem#2886

Styles look shifted.

Taxes - provide amount - placeholder text missing

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2907

• placeholder text missing

Accessibility - Address block - not reading button text

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2847

This address is links:

• In the US
• APO/FPO/DPO
• Outside the US

All 3 buttons are not reading the text, instead are reading brackets and other code

• Apt line is also doing the same thing

Can't scroll down to fully see Yes/No buttons for "I have read instructions..." modal in IE11

From e-QIP-prototype-truetandem created by ryanhofdotgov : ryanhofdotgov/e-QIP-prototype-truetandem#2923

See attached screenshot with IE11. This is under 0.5.2.

Data loading question

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2881

• Can we not show the data loading screen/animation when there is no data to load?

DISABLE_2FA defaulting to false creating confusion during SAML setups

From e-QIP-prototype-truetandem created by ryanhofdotgov : ryanhofdotgov/e-QIP-prototype-truetandem#2862

Revisit role of the DISABLE_2FA configuration option. It currently defaults to false. Recent tests with MITRE seem to indicate it interacts oddly with the SAML option; i.e. application was showing 2FA screen during a semi-operational/semi-broken SAML authentication flow.

As eApp is only going to be deployed in production with SAML option, we should revisit the eApp's defaults. One possible course of action:

• remove DISABLE_2FA and add ENABLE_2FA instead (default to false)

Mental health - diagnosis - fields not saving - Data persistence

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2918

Fields that are not being saved:
Health care professional info

• Name
• Phone
• Address

Agency

• Name
• Phone
• Address

Update project to use Snyk

Gemnasium.com is shutting down, 18F handbook recommends Snyk for Go and Javascript:
https://before-you-ship.18f.gov/security/static-analysis/

Also see example in Forest Service project:
USDAForestService/fs-open-forest-platform#166

Accessibility - address verification modal - disable form

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2848

When verification modal is up can we disable the rest of the form?

Users can tab through the rest of the form without interacting with the address verification modal

Employment activities - phone number text

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2872

Other federal employment branch - outside US address
Stray text in phone number label

Adding attachments - suggested icon change

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2940

Users were confused with the "Other" option icon. They thought it was email.

• Suggest changing to a generic file icon or mailbox

Not all checkboxes not working

Checkbox fails to check/uncheck for the SSN "Not applicable" and the date "Estimated" elements.

Your history accordion item numbers

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2889

Accordion item numbers are missing for:

• Where you have lived
• Employment activites
• Where you went to school

Address block - international address - 2 comments

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2867

• After switching to international address an additional comment link was added (this was on schools page)

Move to a new CI/CD system

This repository has been using Travis CI, however access to Travis CI is now turning off. When you need to use this repository again, convert it to a CI/CD system which is in the ITSP such as Circle CI.

Police record - offenses - Error showing but no errors

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2913

Error showing on review screen but no errors are there

Submit screen - button arrows spacing

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2857

• Need more space between arrows and text in buttons - add 1rem

Current:

Desired:

Your history - employment - incident blocks margin

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2855

• Add 3rem margin above "Provide the date you..."

Current:

Desired:

Accessibility - submit screen

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2852

On the submit screen -

tab order is broken - tabbing out of the first signature takes the user the top of the page (to the skip to main content link)

Note: made this one a high because it would prevent a user from submitting the form

Modals - User can tab behind the instructions/agreement - focus

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2926

Users can tab through the content behind the instructions/agreement

Seen on Chrome, Edge, IE 11

Relationships - marital - location of marriage not saving - Data Persistence

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2916

• Marriage location not saving

substance use - counseling - missing error message

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2908

• when present is selected on the date there is an error

Accessibility - required fields

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2892

• Need to announce all required fields (asterisks) throughout form - alt text stating "Required"

Context from Audit:

Add validation of email domains

As an application reviewer, I want the applicants to receive immediate feedback when they mistype the domain of an email address in the form, so that I can receive more accurate information.

As a user, I want to the ability to submit the provided email address regardless of whether eApp believes it to be incorrect.

The system should validate the FQDN of the email address via a validation service or DNS lookup, something similar to:
https://www.mailgun.com/email-validation
The form should degrade gracefully if the email domain validation service is not available.

Marital - phone number - "use my current phone number" button

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2868

Looks like we are missing the "Use my current telephone number" button in the marriage, civil union, domestic partnership, and separated branches

This would be a text button and pull in current phone number (like the current address button)

Main navigation - accessibility

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2927

Nav items with children are not clear to screen readers.

• Can we call these out? Make them say "Open to see subsections"

Document Adding A New Feature to the Prototype

From e-QIP-prototype-truetandem created by pjsumm : ryanhofdotgov/e-QIP-prototype-truetandem#304

As an eApp developer I want to document how a new developer can contribute a new feature to the eApp prototype so that new developers can follow that pattern in the future to easily contribute new features without guidance.

Acceptance Criteria

• Document that explains how to create a new feature in the eApp prototype SF86 form
• The document has been evaluated and reviewed by the 18f PMO
• The document has been approved by the 18f PMO

IE 11 and Edge nav scrolling issue

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2837

• On shorter screens (with less content) the nav isn't responsive when trying to scroll down. It jumps up and down slightly but doesn't scroll down. Sometimes eventually it unlocks and scrolls

Data persistence additional test automation

From e-QIP-prototype-truetandem created by pjsumm : ryanhofdotgov/e-QIP-prototype-truetandem#2879

There should be automated integration tests that exercise the API calls for persisting and reading back form data (Save, AllSections, Section). Data can be verified by db introspection directly, or via an API call. Test data should cover 90% (ideally all) sections/fields of the SF-86 (via one large/complete application or smaller test snippets, as appropriate). There is likely test data overlap between these tests and the one John is putting together for eQIP API submission.

The automated test priority is to execution pathways that are likely to lead to cause of errors seen by Paul, or were problems historically.

Update deploy mechanism to use refresh token instead of command-line credentials

Current deploy script (bin/deploy-cloudgov) passes Cloud Foundry credentials as command-line parameters to docker run command. Better approach is to use ~/.cf/config.json – see:
cloudfoundry/cli#919

Mitigated as build occurs in a private, virtualized environment – no outside processes can inspect command-line arguments of running processes.

Your history- where you went to school - person who knows you email

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2856

Person who knows you branch>

• Email option should say "I don't know" instead of "Not applicable"

Possible authn error due to client clock skew on IE

From e-QIP-prototype-truetandem created by ryanhofdotgov : ryanhofdotgov/e-QIP-prototype-truetandem#2902

In 78a5316, a cookie is set with an expires value one minute into future, based on server time. Themax-age value is set to 60 seconds.

Historically, all the common browsers except IE respected max-age, with IE only supporting expires. If that is still the case, if the user's workstation's clock is incorrect, then the cookie logic may not work as expected.

Test IE11 with a clock set an hour behind and verify.

I did find this Stackoverflow discussion which indicates that IE11 may now respect the expires parameter on the cookie.

If the test passes under newer versions of IE11 but not older ones, it still may be worthwhile to implement alternative logic that is not dependent on the client's clock.

Cohabitant - accordion item showing error

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2906

• Cohabitant accordion item is showing an error on review screen when there are no errors

Marital - date of separation bug

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2905

Error being thrown in marital section when there is no DOB error

Where you have lived - person who knows you phone block update

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2870

• Remove time of day and phone number type from all 3 numbers

• Also swap the order here, lets do:

• Cell phone first

• Evening

• Day

Current:

Desired:

saml.go:SamlCallbackHandler should be validating NameID, not `cleaning`

From e-QIP-prototype-truetandem created by ryanhofdotgov : ryanhofdotgov/e-QIP-prototype-truetandem#2864

SamlCallbackHandler calls cleanName on the NameID – it should be validating the name instead. If there is unexpected content, the handler should just fail.

For our purposes, you could restrict it narrowly to whatever is allowed in a CN of a x509 certificate or an email address and the max length of the column in PostgreSQL. (Though in general, a SAML consumer should accept whatever the SAML assertion XSD said was OK for a NameID element).

The trouble with "cleaning" the NameID and truncating it before looking it up in the db, is that hypothetically that would map multiple users to the same account. For example:
a) Ryan<insert 196 characters here>1
b) Ryan<insert same 196 characters here>2
These would be mapped by cleanName to the same username.

There are similar hypothetical issues with removing the whitespace characters. (Tangential, but if you were needing to clean, you would also need to search for all the Unicode whitespace/non-printable characters/etc. since we are dealing with UTF-8 vs ASCII.)

Review STIG checklist comments

From e-QIP-prototype-truetandem created by pjsumm : ryanhofdotgov/e-QIP-prototype-truetandem#2869

As an eApp stakeholder I want to review comments on the eApp STIG checklist provided by the eApp security team and respond to the comments so that I can return updated comments to the security team for further review.

Acceptance Criteria:

• Provide responses to ~50 security team questions
• Facilitate feedback from infrastructure team
• Submit result to eApp security team

Other name used validation - error on birthday

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2866

Currently an error is thrown when the from date is the same as DOB.

For a nickname used since birth this would be a valid answer.

• Can we not thrown an error on DOB but only throw error on date before DOB?

Previously used APO address - not saving APO selection

From e-QIP-prototype-truetandem created by Jay-Neighbours-I72 : ryanhofdotgov/e-QIP-prototype-truetandem#2871

• When using a Previously used APO address the initial selection of APO/FPO/DPO didn't come through.
• Can't reselect the APO/FPO/DPO button

Vertical scrollbar in IE11 clips Logout and other visual elements

From e-QIP-prototype-truetandem created by ryanhofdotgov : ryanhofdotgov/e-QIP-prototype-truetandem#2924

In IE11, when the vertical scrollbar is present (which is most of the time), the Logout link at the top and other elements on the page are rendered underneath it. Looking at another site (e.g., news.google.com in IE11) I don't see a similar issue.