usdigitalresponse / arpa-reporter Goto Github PK

From: [email protected]

blah test test

https://www.jotform.com/uploads/Huang_Mindy/212695501572052/5293998396511351146/Input%20template%201.png

this logic is still buggy. in particular, we only seem to set viewPeriodId in loadApplicationSettings and it ends up being null a lot of the time, which kinda still works because it should be set to the current period, and the backend generally also uses the current period if the specified period is not set, but this is not ideal. the entire flow of how the app loads it's initial state into the store should be re-worked.

Originally posted by @igor47 in #233 (comment)

Actions:

• Reporting Periods > Certify the current reporting cycle

Expected result:

• Once it's finished certifying, the previous reporting cycle should fill in its "Certified At" Column

Actual result:

• the previous reporting cycle's "Certified At" Column stays blank until user refreshes the page

we still need to get clarity on this.

both a feature request and possibly a bug (because possibly, reports that include previously-reported projects/recipients would be rejected by treasury at the bulk upload stage)

blah test test

https://www.jotform.com/uploads/Huang_Mindy/212695501572052/5293998396511351146/Input%20template%201.png

partners would love to know which projects came from which uploads -- we need an association table and a ui to view it.

The error is "The file [filename] already exists."

From: [email protected]
Urgent

blah test test

https://www.jotform.com/uploads/Huang_Mindy/212695501572052/5293998396511351146/Input%20template%201.png

@mhuang-usdr
test test

It's easy to trigger confusing errors if you accidentally forget to await a promise.

It would be great if we can work out how to lint for this — hopefully we don't have to migrate the entire codebase to TypeScript to achieve it.

the data in the DB supersedes the data in the sheets, as it may have been edited.

an interesting question is, which sub-recipients are going into the export. maybe any that haven't been reported yet? we may add a column to the table to indicate that it was used. alternatively -- they ARE linked to their upload, so maybe we'll pull all subrecipients that came from all uploads that are valid... no, that won't work, it's not just valid, it's ALL uploads from the given reporting period.

depends on #268

we discussed the issue here: https://usdigitalresponse.slack.com/archives/C031R1Y49KL/p1650992016937089

saw it again this morning. here are some relevant log snippets (i removed the non-relevant lines):

May 12 08:09:01 AM  Rewriting GET /login to /index.html
May 12 08:09:01 AM  ::ffff:10.207.217.69 - - [12/May/2022:15:09:01 +0000] "GET /login HTTP/1.1" 200 1485
May 12 08:09:02 AM  ::ffff:10.206.27.224 - - [12/May/2022:15:09:02 +0000] "GET /api/health HTTP/1.1" 200 16
May 12 08:09:02 AM  ::ffff:10.207.209.69 - - [12/May/2022:15:09:02 +0000] "GET /css/app.edc339d9.css HTTP/1.1" 200 2733
May 12 08:09:02 AM  ::ffff:10.207.217.69 - - [12/May/2022:15:09:02 +0000] "GET /js/app.3d966fd5.js HTTP/1.1" 200 69045
May 12 08:09:02 AM  ::ffff:10.207.237.63 - - [12/May/2022:15:09:02 +0000] "GET /js/chunk-vendors.cb5c7011.js HTTP/1.1" 200 576845
May 12 08:09:02 AM  (node:70) UnhandledPromiseRejectionWarning: Error: select "users"."id", "users"."email", "users"."role", "users"."agency_id", "users"."tags", "roles"."rules" from "users" inner join "roles" on "roles"."name" = "users"."role" where "users"."id" = $1 - Connection terminated unexpectedly    at Connection.<anonymous> (/opt/render/project/src/node_modules/pg/lib/client.js:132:73)    at Object.onceWrapper (events.js:482:28)    at Connection.emit (events.js:376:20)    at Connection.emit (domain.js:470:12)    at Socket.<anonymous> (/opt/render/project/src/node_modules/pg/lib/connection.js:57:12)    at Socket.emit (events.js:376:20)    at Socket.emit (domain.js:470:12)    at TCP.<anonymous> (net.js:673:12)
May 12 08:09:02 AM  (node:70) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 16)
May 12 08:09:02 AM  Connection Error: Connection ended unexpectedly
May 12 08:09:02 AM  ::ffff:10.207.10.253 - - [12/May/2022:15:09:02 +0000] "GET /favicon.ico HTTP/1.1" 200 4286
May 12 08:09:07 AM  ::ffff:10.206.27.224 - - [12/May/2022:15:09:07 +0000] "GET /api/health HTTP/1.1" 200 16
May 12 08:09:12 AM  ::ffff:10.206.27.224 - - [12/May/2022:15:09:12 +0000] "GET /api/health HTTP/1.1" 200 16
May 12 08:09:14 AM  ::ffff:10.207.10.253 - - [12/May/2022:15:09:14 +0000] "GET / HTTP/1.1" 200 1485

May 12 08:10:17 AM  Rewriting GET /login to /index.html
May 12 08:10:17 AM  ::ffff:10.207.237.63 - - [12/May/2022:15:10:17 +0000] "GET /login HTTP/1.1" 200 1485
May 12 08:10:18 AM  ::ffff:10.207.10.253 - - [12/May/2022:15:10:18 +0000] "GET /favicon.ico HTTP/1.1" 200 4286

May 12 08:12:09 AM  Connection Error: Connection ended unexpectedly
May 12 08:12:09 AM  ::ffff:10.207.209.69 - - [12/May/2022:15:12:09 +0000] "GET /api/sessions HTTP/1.1" 200 101
May 12 08:12:12 AM  ::ffff:10.206.27.224 - - [12/May/2022:15:12:12 +0000] "GET /api/health HTTP/1.1" 200 16
May 12 08:12:15 AM  ::ffff:10.207.209.69 - - [12/May/2022:15:12:15 +0000] "GET /api/sessions HTTP/1.1" 200 101
May 12 08:12:16 AM  ::ffff:10.207.237.63 - - [12/May/2022:15:12:16 +0000] "GET /api/uploads HTTP/1.1" 200 14
May 12 08:12:16 AM  ::ffff:10.207.237.63 - - [12/May/2022:15:12:16 +0000] "GET /api/projects HTTP/1.1" 200 15
May 12 08:12:16 AM  ::ffff:10.207.217.69 - - [12/May/2022:15:12:16 +0000] "GET /api/configuration HTTP/1.1" 200 30649
May 12 08:12:16 AM  ::ffff:10.207.217.69 - - [12/May/2022:15:12:16 +0000] "GET /api/reporting_periods HTTP/1.1" 200 12409
May 12 08:12:16 AM  ::ffff:10.207.10.253 - - [12/May/2022:15:12:16 +0000] "GET /api/agencies HTTP/1.1" 200 11223
May 12 08:12:16 AM  ::ffff:10.207.10.253 - - [12/May/2022:15:12:16 +0000] "GET /api/subrecipients HTTP/1.1" 200 20
May 12 08:12:16 AM  ::ffff:10.207.209.69 - - [12/May/2022:15:12:16 +0000] "GET /api/application_settings HTTP/1.1" 200 617

May 12 08:13:18 AM  ::ffff:10.207.217.69 - - [12/May/2022:15:13:18 +0000] "GET /api/sessions HTTP/1.1" 200 101
May 12 08:13:18 AM  ::ffff:10.207.217.69 - - [12/May/2022:15:13:18 +0000] "GET /api/configuration HTTP/1.1" 200 30649
May 12 08:13:18 AM  ::ffff:10.207.209.69 - - [12/May/2022:15:13:18 +0000] "GET /api/agencies HTTP/1.1" 200 11223
May 12 08:13:18 AM  ::ffff:10.207.217.69 - - [12/May/2022:15:13:18 +0000] "GET /api/application_settings HTTP/1.1" 200 617
May 12 08:13:18 AM  ::ffff:10.207.209.69 - - [12/May/2022:15:13:18 +0000] "GET /api/uploads HTTP/1.1" 200 14
May 12 08:13:19 AM  ::ffff:10.207.209.69 - - [12/May/2022:15:13:19 +0000] "GET /api/reporting_periods HTTP/1.1" 200 12409
May 12 08:13:19 AM  ::ffff:10.207.209.69 - - [12/May/2022:15:13:19 +0000] "GET /api/subrecipients HTTP/1.1" 200 20
May 12 08:13:19 AM  ::ffff:10.207.209.69 - - [12/May/2022:15:13:19 +0000] "GET /api/projects HTTP/1.1" 200 15

May 12 08:14:10 AM  Rewriting GET /login to /index.html

May 12 08:15:41 AM  ::ffff:10.207.209.69 - - [12/May/2022:15:15:41 +0000] "GET /api/sessions HTTP/1.1" 200 101
May 12 08:15:41 AM  ::ffff:10.207.10.253 - - [12/May/2022:15:15:41 +0000] "GET /api/projects HTTP/1.1" 200 15
May 12 08:15:41 AM  ::ffff:10.207.209.69 - - [12/May/2022:15:15:41 +0000] "GET /api/uploads HTTP/1.1" 200 14
May 12 08:15:41 AM  ::ffff:10.207.217.69 - - [12/May/2022:15:15:41 +0000] "GET /api/application_settings HTTP/1.1" 200 617
May 12 08:15:41 AM  ::ffff:10.207.10.253 - - [12/May/2022:15:15:41 +0000] "GET /api/reporting_periods HTTP/1.1" 200 12409
May 12 08:15:41 AM  ::ffff:10.207.237.63 - - [12/May/2022:15:15:41 +0000] "GET /api/configuration HTTP/1.1" 200 30649
May 12 08:15:41 AM  ::ffff:10.207.10.253 - - [12/May/2022:15:15:41 +0000] "GET /api/subrecipients HTTP/1.1" 200 20
May 12 08:15:41 AM  ::ffff:10.207.237.63 - - [12/May/2022:15:15:41 +0000] "GET /api/agencies HTTP/1.1" 200 11223

i'm going to guess that the initial traceback from knex/postgres is related to knex/knex#3523 where someone says:

One can probably work around this issue by setting minimum pool size to 0 and set idle timeout to be like 5 minutes or so. In that way connections will be discarded from pool before that.

and, googling for the "connection ended unexpectedly" error, i find strapi/strapi#8117 where the author also suggests messing with connection pool settings to work around the problem.

via @lhitchon , db migrations on live instances are done by setting local env vars on a developer's laptop to the postgres instance creds taken from a particular render instance, and then the migration script is invoked manually on the laptop. this should be documented in the readme

After May 26th we will have active users. We should reconsider our automatic deployment process.

I propose considering something like the following:

• Keep the USDR staging deployment pointed to the staging branch
• Reset the main branch to latest commit of staging
• Point RI and OH branches to main branch
• Continue to open PRs against staging branch
• Manually advance main after communicating new set of changes to users

Returns a .json file

The helper function log is defined separately in many source files on the server. We should just extract that to a single definition.

let log = () => {}
if (process.env.VERBOSE) {
  log = console.log
}

Expected export templates are here: https://drive.google.com/drive/u/0/folders/1Z-AS0acaY_HdO5Y4rbFNwBTOuxeMTsjm

Mapping from project expenditure category to templates starts on page 62: https://drive.google.com/file/d/1rNgA8bAUhUThJfzjKPJzKzYggOFp6dtt/view?usp=sharing

Supported templates:

• expendituresGT50000BulkUpload.xlsx
• expendituresLT50000BulkUpload.xlsx
• paymentsIndividualsLT50000BulkUpload.xlsx
• project18_229233BulkUploads.xlsx
• project19_234BulkUploads.xlsx
• project2128BulkUploads.xlsx
• project214_224227BulkUploads.xlsx
• project236BulkUploads.xlsx
• project31BulkUpload.xlsx
• project32BulkUpload.xlsx
• project4142BulkUpload.xlsx
• project51518BulkUpload.xlsx
• project519521BulkUpload.xlsx
• projectBaselineBulkUpload.xlsx
• subRecipientBulkUpload.xlsx
• subawardBulkUpload.xlsx

From: [email protected]

blah test test

https://www.jotform.com/uploads/Huang_Mindy/212695501572052/5293998396511351146/Input%20template%201.png

we got to a bunch of lint errors in the repo, fixed in #74 . this is probably because we're not automatically running lint on PRs/blocking PRs from being merged due to lint errors. could be something to look into

Go to Reporting Periods > Edit a reporting period.

The button for "Update Reporting Period" is missing a space.

• Email language
• DOM<title>
• Add "ARPA Reporter" to the page title, so "Rhode Island" becomes "Rhode Island - ARPA reporting"

From:

details

https://www.jotform.com/uploads/Huang_Mindy/212695501572052/5085800730197978466/Santa%20Clara%20hail.jpeg

Urgent - actively blocking work

test test

From: [email protected]

Upon a brainstorming session with St. Louis, they had this idea:

"You can easily add KPI tracking as another tab to the spreadsheet. There's no aggregation or validation needed since KPIs differ by department. You just need to be able to output it in the right format for Treasury at the end"

Blank entries in reporting period menu due to dependence on period.id being sequential from 0

we should just keep track of roles in the store directly, vs as the only remaining child of the configuration object.

Originally posted by @igor47 in #314 (comment)

It doesn't appear that the ARPA tool is always correctly identifying the most recently validated upload for use in exports.

From:

details

https://www.jotform.com/uploads/Huang_Mindy/212695501572052/5085800730197978466/Santa%20Clara%20hail.jpeg

On both the Uploads and Upload screen, users have reported not being able to find the download button.

Maybe we can make this more visible, or consider experimenting with a different icon?

maybe alias them? get is kinda hard to grep for, compared to, like, getReportingPeriod

Originally posted by @igor47 in #317 (comment)

the node version to use with the project is unclear. we should add guidance about that to the readme and maybe a file used by nodenv to make it more clear. we might want to upgrade the nodejs version. we should coordinate the upgrade with the various render instances that run this code -- there might be render configuration that specifies the version.

Maybe we can just support a typed filter, instead of populating a very large dropdown?

Feedback from Ozzie:

New ‘Reporting Period’ Validation Error: I am receiving new validation errors for files that previously were accepted by Render for the reporting period. The error looks like this:

In this instance, I confirmed that the file/record I submitted does have a reporting period starting 1/1/22 and ending 3/31/22 on tab ‘Cover’ row 2 column 5 and 6.

I suspect this has something to do with the changes I made yesterday to support preserving dates in exports.

Let's aim for parity with CARES reporter:

• First upload that specifies a Subrecipient is captured
• Duplicate Subrecipients in subsequent reports are ignored
• (Optional) Report simplified "warnings" during the upload process

We know the currently-selected reporting period at any given time in the tool. This should allow us to write the reporting period dates into the template as protected fields. This could reduce confusion for agency users.

I think I'd like to make this a POST request, for clarity.

Originally posted by @ajhyndman in #207 (comment)

A lot of this data is output from other software systems and has unnecessary sub-cent precision.

We normalise casing during login, but not on account creation!

What do you think about renaming this to exportedForReportingPeriod? I was a little confused at first because I thought this function returned "all" valid uploads for the reporting period.

Originally posted by @ajhyndman in #311 (comment)

revalidate Error: {"error":"update \"uploads\" set \"validated_at\" = $1, \"validated_by\" = $2 where \"id\" = $3 returning * - current transaction is aborted, commands ignored until end of transaction block"}

This may also have been resolved with recent changes, but we should re-check after #282 lands.

Currently blocked on getting templates with data.

• Revise subrecipient db table
• Push subrecipients into DB on write
• Do something about conflicts (?)

I used to be able to update these fields (made changes from Quarterly # to align with reporting period end dates) but now am not able to! Thanks!

Put a link to the upload details page in arpa-reporter directly in the audit report!