leijurv / gb Goto Github PK

2022/01/03 09:02:35 open /root/skycacherestore/skycache-1640113240/51716,27413.zst for write
panic: open /root/skycacherestore/skycache-1640113240/51716,27413.zst: too many open files

goroutine 1 [running]:
github.com/leijurv/gb/download.execute(0xc000bd5660, 0x20, 0x20, 0x0, 0xc02794b9b0, 0xc02d57b6b0, 0xc02794b9e0)
	github.com/leijurv/gb/download/restore.go:273 +0xc93
github.com/leijurv/gb/download.Restore(0xc010a3dff0, 0xf, 0xc0001263d8, 0x15, 0x0)
	github.com/leijurv/gb/download/restore.go:249 +0x16f9
main.main.func16(0xc0000ea160, 0xc00010e300, 0xc00010e330)
	github.com/leijurv/gb/main.go:302 +0xab
github.com/urfave/cli.HandleAction(0xec66e0, 0x109f0e0, 0xc0000ea160, 0xc0000ea160, 0x0)
	github.com/urfave/[email protected]/app.go:524 +0x105
github.com/urfave/cli.Command.Run(0x10302f9, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x103fa2c, 0x15, 0x0, ...)
	github.com/urfave/[email protected]/command.go:173 +0x579
github.com/urfave/cli.(*App).Run(0xc0001ad500, 0xc0000320c0, 0x4, 0x4, 0x0, 0x0)
	github.com/urfave/[email protected]/app.go:277 +0x808
main.main()
	github.com/leijurv/gb/main.go:373 +0x1385

the entire restore was expected to be about 1.8 million files

Could put something between the open file and the compressor that records the quantity of bytes read, and compares it to the expected size from stat-ing the file.

something like this could be cool.

Then gb would actually be able to report what percent done it is instead of just Currently uploading: /path/to/some/huge/linux.iso over and over and you having no idea how much progress it's made through that file

This file causes a panic while running a backup:

-rw-r--r-- 17 root root 213K Dec 31  1969 /path/to/SVI_0001.JPG

fileHasKnownData panics with "CHECK constraint failed: files" because of the constraint on the files table, CHECK(fs_modified >= 0).

I can recreate the table without the constraint, but perhaps that constraint should be reconsidered, or better error reporting should be added since there's no indication of which file failed or why.

There are a lot of pending changes on the google doc.

I've been experimenting with gb on a server with 2GB of RAM and 3.5GB of swap. It mostly works well, but BackupDB() causes a panic when memory can no longer be allocated, either during compression or encryption.

I was able get it working for a bit longer by switching from github.com/DataDog/zstd to tailscale.com/smallzstd and disabling the decryption and decompression verification, but with the SQLite database now at around 1GB, I'm back to running out of memory.

Backing up the database without having to read the whole file, and then compress it and encrypt it in memory, would be useful.

122a2d0

gb no longer builds on mac osx

 Lurfs-MacBook-Pro:gb leijurv$ go build
# bazil.org/fuse
../go/pkg/mod/bazil.org/[email protected]/error_std.go:26:20: undefined: errNoXattr
../go/pkg/mod/bazil.org/[email protected]/fuse.go:171:12: undefined: mount
../go/pkg/mod/bazil.org/[email protected]/fuse.go:236:24: undefined: maxWrite
../go/pkg/mod/bazil.org/[email protected]/fuse.go:425:32: undefined: maxWrite
../go/pkg/mod/bazil.org/[email protected]/fuse.go:810:12: undefined: openFlags
../go/pkg/mod/bazil.org/[email protected]/fuse.go:828:18: undefined: openFlags
../go/pkg/mod/bazil.org/[email protected]/fuse.go:845:18: undefined: openFlags
../go/pkg/mod/bazil.org/[email protected]/fuse.go:868:18: undefined: openFlags
../go/pkg/mod/bazil.org/[email protected]/fuse.go:991:12: undefined: openFlags
../go/pkg/mod/bazil.org/[email protected]/fuse.go:1413:20: undefined: maxWrite
../go/pkg/mod/bazil.org/[email protected]/fuse.go:1413:20: too many errors
 Lurfs-MacBook-Pro:gb leijurv$ go version
go version go1.19 darwin/arm64
 Lurfs-MacBook-Pro:gb leijurv$ 

@babbaj

When trying to restore a directory with a non-null files.end, it is pretty much a certainty that files.end < now. I don't understand why gb fails to restore instead of just restoring latest version?
This issue is compounded because as far as I can tell there's not a clear way to get the latest timestamp for a directory? I had to gb ls and grab a random one from a file, though this may just be me missing something.

To resolve, I think:

• gb restore should not exclude files where files.end < now in the query
• gb ls should also output information about directories (though this seems much more complicated)

What do you think?

Certainly! Here's a template for a GitHub issue that you can use to explain the merits of Rust and try to convince the author to rewrite their program in Rust:

Title: Consider Rewriting [Program Name] in Rust for Enhanced Performance and Safety

Issue Body:

Hello [Author's Name],

I hope you're doing well! I'm a dedicated user of your [Program Name], and I've been thinking about ways to improve its performance, reliability, and maintainability. After careful consideration, I'd like to propose the idea of rewriting the entire codebase in Rust. Here are some compelling reasons why I believe this would be a valuable endeavor:

1. Memory Safety: Rust's ownership system and borrowing rules make it nearly impossible to encounter null pointer dereferences, buffer overflows, or data races. This translates to a more robust and secure application, reducing the risk of critical security vulnerabilities.

2. Concurrency: Rust has built-in support for concurrent programming with its ownership model and async/await syntax. This could greatly improve the program's scalability and responsiveness, especially if it involves I/O-bound operations or needs to utilize multiple CPU cores efficiently.

3. Performance: Rust offers predictable and efficient memory management, which can result in better runtime performance compared to languages like Go. This can be particularly beneficial for CPU-intensive workloads.

4. Ecosystem: Rust has a growing and vibrant ecosystem of libraries and tools, making it easier to find and integrate dependencies to enhance the functionality of [Program Name]. Additionally, the Rust community is known for its commitment to documentation and best practices.

5. Long-term Maintainability: Rust's focus on code correctness and expressive type system can lead to more maintainable code over time. This can make it easier for both current and future maintainers to understand and extend [Program Name].

6. Cross-platform Compatibility: Rust's support for cross-compilation allows you to target a wide range of platforms without sacrificing performance or safety.

7. Community Support: The Rust community is known for its helpfulness and inclusivity. If you decide to undertake this transition, you'll likely find a wealth of resources and assistance.

I understand that a complete rewrite is a significant undertaking, but I believe the benefits could be substantial. It's worth noting that some prominent projects have successfully migrated to Rust, such as Servo (a web browser engine) and parts of the Firefox browser.

I'd be more than willing to contribute to this effort in any way I can, whether it's by helping with the codebase, documentation, or testing. I'd love to hear your thoughts on this proposal and whether you'd be open to discussing the possibility of transitioning [Program Name] to Rust.

Thank you for your time and consideration.

Best regards,
[Your Name]

Feel free to customize the template to fit your specific situation and relationship with the author. It's important to be respectful and open to a discussion, as a complete rewrite is a significant decision for any project.

I like where you are going with this backup solution. Though I know one of the benefits of gdrive is that it gives me a complimentary backup to any NSA data center, I'd be more interested in using the plethora of free, high quality space that the internet already offers - like DNS resolver caches, forum avatars, Discord attachments, Minecraft skins, other people's git repos, and exposed Zookeeper znodes.

Okay, jokes aside, I am wondering if this program can fit my use case. I am a low tier cheapskate who uses Wasabi for storage. Turns out that their data centers are a bit less reliable than I had hoped, but I still think $5/TB is a great deal, so I want to spread my storage across buckets in different regions without incurring significant cost with full 1:1 mirrors. Currently I use Synology Hyper Backup to backup my NAS to S3, which is great, but the format is completely opaque and I'd rather use FOSS tools.

I wonder how feasible it would be to distribute the backup blobs across multiple buckets or targets, and then track the storage location in the database.

Danke.

It looks like the SQLite database isn't getting checkpointed during a run of gb backup.

The WAL is growing to many times the ultimate size of the database, and the database isn't getting checkpointed until the backup completes. During a long backup, the WAL grew for my database grew to 45GB; the database ended up being about 5GB.