ryanscottlewis / cutback Goto Github PK
View Code? Open in Web Editor NEWBackup system with the goal of creating smaller, more focused backups.
Home Page: https://ryanscottlewis.github.io/cutback
License: MIT License
Backup system with the goal of creating smaller, more focused backups.
Home Page: https://ryanscottlewis.github.io/cutback
License: MIT License
Cutback only works on POSIX-compliant operating systems, but once there is option interpolation (#5) and the reliance of piping/redirection is broken (possibly based on Cutback determining if the operating system is POSIX or not), then we should be able to have Cutback be entirely cross-platform. This also obviously depends on valid Crystal targets (windows??).
Since the business end of Cutback really only runs system commands, a backup configuration should be able to generate a runnable script which runs the same commands.
Generated script could even re-implement the identifier generation functionality of cutback to continuously generate backups with incremental indicies.
--uuid
option or combine with #5 and allow an interpolation option --identifier
; replacing --index
and --date
Right now, only Cutback errors are handled (naively). All errors should be handled and the verbosity of the configuration should denote whether to show a simple message or a full error message + backtrace (for issue reporting)
Possibly related to #1
When collecting records, Cutback should be able to optionally gather metadata about each record item including file size, *nix permissions/times, exif/filetype specific metadata.
Show a warning when this gathered metadata exceeds size of the actual data, causing the act of gathering the record of that file useless (probably a rare instance but totally possible).
If this process is also applied to manifest items, the FileListCalculator
class becomes near-obsolete as we would already have that data in most cases. The only time it would be used is when a user chooses both to view progress and not generate the manifest item attributes. In this case, the user should be warned that "hey, we're already generating this data, you might as well save it)
So far, this application creates backups, but it should also list, read, update, and destroy backups as well (CRUD).
Features:
Allow a user to output to multiple locations. for example I might want a copy of the final backup to be in both Google Drive as well as a RAID 1 NAS.
Use Process
to run the tool to get the process id and report during/after build?
/proc/PID/io
/proc/PID/status
Auto-remove oldest after certain backup count threshold
Warn if threshold is larger than the output directory can hold?
For example if the last archive was 30GB and the output dir can hold 1.2TB then it can hold 40 backups. If the backup threshold is at 50 then the output directory will run out of space before that threshold is ever met.
Right now, files are matched to be listed in Manifest or Records by filename patterns only. It would be nice to filter by file size or combine rules such as all files less than 100MB that matches the given pattern.
Filters:
/foo/bar/**/*
An option should allow the user to be user that they will not be prompted for any warning/questions when generating a backup.
Search order:
--config
)Note that this is semi-implemented within
options_parser.cr
however it is not documented anywhere.
Right now, we use predetermined tools and we assume any tool passed to the option works like
the default tool for that option. Users should be able to set entire commands, although some commands are quite complicated (see find
or compress
).
These would require interpolations within themselves, for example xz
needs -T #{@paths.manifest}
so it could do something like -T %{manifest}
or -T $(manifest)
.
This would enable users to use fzf
instead of find
for the search step, for example, which at the moment would not work at all since fzf
's options are wildly different.
This causes Command
and it's subclasses to be obsolete, allowing us to rename Action
to Command
to align more with the command pattern as laid out in #20.
An optional step for encrypting the archive.
Should default to being performed after compression with the option be perform before compression.
embed/i18n.yml
This would take a whole lot of community effort, possibly through a 3rd party translation service.
Even if only a manifest or records file is generated, it's nice to see the file size totals for the files within them. Same goes for most other information within the manifest.
Validate tools before usage and either raise error or allow for continuation (for optional steps, if allowed by the configuration).
app.yml
, written in Markdown)When extracting the package (built from make package
), it's contents are dumped into pwd
/output path but it would be nicer (and is conventional) to have a single entry within a package, named after the version.
This would change the manual install steps:
tar xf cutback-linux-x86_64-0.0.1-pre.tar.gz
cd cutback-0.0.1-pre # The new bit
make install
Several actions/commands are strung together which form a logical "group" of actions.
Perhaps Cutback could handle both of these cases by splitting actions/commands into stages (in practice rather than theory which is currently the case) and allowing users to move actions to wherever they want within these stages.
Some options such as date
and index
don't really make sense for inclusion within the config files. Add a config
boolean for options within app.yml
which indicates if the option is also a config file variable.
At the moment, there is only a placeholder for this file.
Similar to #4, the resulting backup should be optionally signed.
Order of operations between encryption and signing needs to be considered:
https://crypto.stackexchange.com/questions/5458/should-we-sign-then-encrypt-or-encrypt-then-sign
Some steps such as compression and encryption can be completely optional.
Since they are optional, they could also be optionally skippable. For example a build configuration might have the compression step enabled, but also skippable if the platform the build is being run on does not have a compressor available.
All internal dependencies are required within setup.cr
. Require at the beginning of files using the dependency? Is this even worth working on?
Instead of incrementing index on each build, it should be incremented with a new option or possibly by passing +
to --index
. This would mean that cutback would operate like a build automation system. Once the build for that index is created, it doesn't rebuild it.
At the moment, the only system package is for pacman on the Arch Linux AUR.
All system packages should be generated via templates and any extra commands needed, for example after generating the PKGBUILD
for the pacman package the following command would need to be called in order to publish to the AUR:
makepkg --printsrcinfo > .SRCINFO
A user might want to try fzf
for finding before falling back to find
when fzf
is not installed.
Use an array for tool options. Might depend on #5.
When a controller delete
action is run, ask for a confirmation of that action, unless the --force
option is given.
To avoid the excessive app.
prefixes within templates.
<%= name %>
instead of <%= app.name %>
.
Tar the final collection of backup files into a single file:
On a side note, all compression and encryption should be optional, regardless of order of operations.
Action
subclasses are simply wrappers to allow a large method to be refactored into multiple smaller methods.
Controller
subclasses contain methods, conventionally called actions
and called such within documentation.
This is confusing from a developer standpoint, if someone mentions an "action", are they talking about an Action
? Or a controller action?
When generating manifest (and possibly records) file(s), a checksum and/or cryptographic hash could be generated for each file entry to be used for checking authenticity and integrity of each file in the backup files archive.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.