richardkruemmel / chat-your-gesetzentwurf Goto Github PK

Description

Implement a scraper service to handle two primary tasks:

1. Database Population:

• Create a script for initial database population from the third-party API.
• Set up logic for incremental updates to the database based on new records from the third-party API.

2. API Monitoring and Database Updating:

• Implement a scheduled task to regularly check the third-party API for updates.
• Develop logic to detect changes in the API data and update the database accordingly.
• Ensure robust error handling and logging for all scraper operations.

Tasks:

• Set up a new directory /backend/app/scraper for the scraper service.
• Implement a script for the initial population of the database.
• Set up logic for incremental updates to the database.
• Create a scheduled task for regular API monitoring.
• Develop change detection logic to identify updates in the API data.
• Implement database update functionality to reflect changes detected in the API data.
• Ensure robust error handling and logging for scraper operations.
• Write tests to verify the correct functionality of the scraper service.
• Document the scraper service functionality, usage, and error handling.

Acceptance Criteria:

• The scraper service should successfully populate the database initially with data from the third-party API.
• The scraper should regularly check the API for updates and apply any updates to the database.
• All scraper operations should have error handling to manage potential issues and should log errors and successful operations for troubleshooting and monitoring purposes.