uschtwill / re-employment-kraken Goto Github PK

Check: https://github.com/apify/crawlee

I suggest Telegram as an alternative way of receiving project alerts for users who are not already using Slack or Notion. The process is quite simple and straight-forward:

• Pick a library providing Telegram bot API access.
• Add config settings for enabling / disabling, Telegram user ID and Telegram bot token.
• Use Telegram bot client to send new projects to configured user.
• Document how to activate Telegram support in README file.

Note: I will provide a pull request for this as I am interested in using this feature.

Requires #8

How to reproduce:

• Configure a search term with many results like java
• Configure plugins freelance-de and freelancer-map-de (could happen with other strategies, too)
• Use a fresh DB by either removing the old one, changing the path or setting DATABASE_ENABLED to false

Observed error:

An unhandled Promise rejection occurred: {
  ok: false,
  error_code: 429,
  description: 'Too Many Requests: retry after 4',
  parameters: { retry_after: 4 }
}

This happens many times. We should think about adding a backoff delay or some kind of rate limiting to prevent this errors. Right now, we have an unhandled promise rejection handler in index.js to help with debugging.

See:

• https://github.com/aceakash/string-similarity (deprecated)
• https://github.com/stephenjjbrown/string-similarity-js

When deploying the application to a server, using a container is preferable since it brings all the required dependencies and runs in isolation.

For this purpose, I suggest the following changes:

• Provide a Dockerfile for packaging the application as a container image.
• Pass .env and database files with Docker volume to preserve them between application runs.
• Provide a Docker compose file as a simple option for running the container with volumes using default paths.
• Document how to build and run the application as a container.

I will provide a pull request for this suggestion soon.

First things first: Nice project, thanks for your effort!

I have seen that the current persistence solution is using plain text files and checking for substrings. While this is good enough for a prototype, it will become a performance issue at some point, given the application is used long enough.

Therefore, I suggest to:

• Include a lightweight embedded database like SQLite.
• Store visited projects in a simple table and set an index on the URL column.
• Retrieve known projects from the database table for determining known projects.
• Remove the text file solution.

If we want to be extra convenient, we could add a migration script for moving existing text file content into the database. However, this might be over-the-top.