ruarxive / apibackuper

Python library and cmd tool to backup API calls

Национальный цифровой архив России / Russian national digital archive (ruarxive.org)

Проект по цифровому сохранению знаний связанных с материалами на русском языке, созданным в России или с Россией связанных (ruarxive.org)

apibackuper's Issues

Add better documentation, more examples, more explanation

• Review several good documents implementations
• Consider one of docs hostings or tools like Docusaurus, Gitbook or readme.io
• Add more documentation and move it to the hosting outside README.md

Support custom code functions and code execution befor, during and after APIBackuper actions.

• Review code and data processing process to identify code application parts. At least add code before running data collection, after page extraction as filters and befor data storage
• Consider best way to add code execution to the project
• Implement code executing
• Add code execution example

Better error handling with custom error handlers

• Review current error handling code
• Add error handling for common errors like ConnectionError exception, error HTTP status pages and so on
• Add more ways on error handling like: retry, ignore, break
• Add options for error handling in config files

Add more "smartness" and API backuping options autodetection.

• Review most common patterns of existing API
• Add automatic detection of API paging and data collection schedule
• Implement automatic API detection

Support extraction of multiple linked objects. For example if there is more than one object to follow or followed objects have multiple data pages.

• Review existing APIs web such behavior
• Consider best way for implementation
• Consider automated data modelling
• Implement code with support of multiple linked objects

Implement database storage instead of ZIP containers for pages and objects

An idea is to store raw pages and collected objects not as JSON/XML files but using embedded DB like SQLite.

• Review SQLite, LevelsDB, RocksDB and other file storages.
• Consider using SQL engine or MongoDB too.
• Choose best storage as default
• Consider implementing multiple db storages
• Implement at least one db storage.
• Linked to implementation of "update" command

Implementation should provide effective storage with builtin compression.

Implement "update" command to extract and update the data from API.

• Implement "update" command with extraction only new data or update records
• Add track changes option, not just add new records, but update changed records too
• Consider keep historical records
• Linked to shifting to database engine from ZIP storage