thatnerduknow / financial-disclosures

Twitter Bot Tracking every Congress members financial disclosure documents freely available through the STOCK act. Follow @CongressionalFD

A progressive Node.js framework for building efficient and scalable server-side applications.

Description

Nest framework TypeScript starter repository.

Installation

$ npm install

Running the app

# development
$ npm run start

# watch mode
$ npm run start:dev

# production mode
$ npm run start:prod

Test

# unit tests
$ npm run test

# e2e tests
$ npm run test:e2e

# test coverage
$ npm run test:cov

Support

Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please read more here.

Stay in touch

• Author - Kamil Myśliwiec
• Website - https://nestjs.com
• Twitter - @nestframework

License

Nest is MIT licensed.

financial-disclosures's Issues

First run tries to open thousands of instances of imagemagick

This is causes by trying to run downloadAndProcessPDF() within records.forEach(). the way it should be done should look something like this

const arr = [1, 2, 3];

await arr.reduce(async (memo, i) => {
	await memo;
	await sleep(10 - i);
	console.log(i);
}, undefined);

// 1
// 2
// 3

console.log("Finished async");
// Finished async

This should only allow one call to downloadAndProcessPDF() at once

Need better tests

The current test suite relies too much on side effects. For example, the test suite for downloadAndProcessPDF() is dependent on both an HTTP request and im.convert() which produces side effects to the file system and is dependent on a working installation of imagemagick and ghostscript which we can't expect a runner like github actions to have in place. methods that have side effects need to be mocked in the tests at the very least with a stub

Implement MongoDB connection instead of using a JSON file to keep track of records

I really wanted to use mongodb from the start but my personal server doesn't support AVX instructions and I was in a hurry so I bodged together a solution backed by a JSON file located in the project directory. I won't be able to host application data on my home server however the pricing for mongodb cloud looks to be fairly sane and considering the amount of data we're talking about here it would be worth implementing it since I have more faith in typeORM than my solution 😄