paddlesteamer / cloudstash Goto Github PK

keep history of hmacs of files against old version attacks

No need to download file everytime fs.Read is called

If updated database upload fails, remove inserted lines and return fuse.EIO

If the remote and local databases are changed at the same time, attempt to merge databases.

• Remove directory doesn't work recursively:

  cloudstash/internal/manager/manager.go

  Lines 239 to 241 in 51107f3

  	// RemoveDirectory removes all of the directory contents and then
  	// removes the directory itself.
  	func (m *Manager) RemoveDirectory(ino int64) error {

• Deleted files don't get deleted in the remote drive.

• When a file is deleted, tracker isn't notified

check if the file is being downloaded already on fuse read

Now StatFS barely returns information about the filesystem:

and get rid of warnings about extended attributes on macOS. A way to do this to create another table with key-value columns and connect it to the files table with a foreign key. Or another way is to add a new xAttr column to the files table and use JSON encoding/decoding.

Add requirements and how to run section

A rwlock is needed for each file opened.

Go over the error messages and

• convert critical errors to panics
• handle the other ones appropriately
  • if a file couldn't be uploaded add it to queue
  • if there is a database error roll back everything

'not found' error implementation is needed in drive/drive.go for the sake of error checks like below:

https://github.com/paddlesteamer/hdn-drv/blob/dbbf5b2957c29ecc7b1bed1e1f4295071d5a102b/manager/manager.go#L71

also, the dropbox implementation should be converted to return 'not found' error where necessary.

Cleanup in handler:

• remove local db copy from host system
• if removed db is different from remote storage, upload it
• unmount fuse fs

The following calls to os.Remove() are not being checked for errors:

• cloudstash/cmd/cloudstash/main.go

  Line 125 in 033e404

  os.Remove(dbPath)

• cloudstash/cmd/cloudstash/main.go

  Line 134 in 033e404

  os.Remove(dbPath)

• cloudstash/cmd/cloudstash/main.go

  Line 140 in 033e404

  os.Remove(dbPath)

• cloudstash/cmd/cloudstash/main.go

  Line 176 in 033e404

  os.Remove(file.Name())

• cloudstash/cmd/cloudstash/main.go

  Line 182 in 033e404

  os.Remove(file.Name())

If a file is opened and then immediately flushed it's not cached but updateMetadataFromCache() is called on it. So we need to keep track of opened files and if any write operation is done on them

before uploading local database file, check whether the remote file is changed. this should also apply for cached files.

If the requested item expires between get and set, the file would be unreachable.
https://github.com/paddlesteamer/hdn-drv/blob/b7e487b49f40fe1a2c5839ee85f46a4d313f9c3a/internal/manager/manager.go#L211-L222

Low probability but can happen. Unfortunately go-cache doesn't support atomic operation for this. There is a PR though:
patrickmn/go-cache#96 but it doesn't look completely safe.

So we can implement that ourselves or risk it and use the code in the PR.

Should we change it to Standard Go Project Layout or should we continue with this old-school, c-like layout?

In the first run, if there is a database file, check if it is a valid database (does it have all tables?)

A validation is required before adding files, creating directories.

don't put plain text passphrase into config file

Now with encryption, it's not easy to check changes with content hash since it's hash of encrypted content.

Need implementation using drive.Drive interface. Also drive.Drive interface may change due to differences between Dropbox and GDrive APIs.

Possible problems:

• Does google drive provide content hash like dropbox?

There should be rwlock in drive.Drive for network operations. For example, an upload operation shouldn't start when there is a download operation processing.

https://github.com/github/super-linter

nautilus (and probably other file explorers) create thumbnails by reading contents of image files which causes the download of all files. it may be fine but also there may be a more efficient solution to this.

https://github.com/paddlesteamer/hdn-drv/blob/dc2e0761208770c7fc80fcd3798f2435c41575f0/cmd/hdn-drv/main.go#L20-L24

Implement a method that checks changes in cached files.

Ideas:

• when a database is updated, compare it with the current database, if a file is different remove it from cache and fs (file hash is required for comparison)
• save hash value in cache among with filename. compare it with hash value in the database. invalidate/validate cache accordingly *****

There are two incomplete go libraries onedrive-go and go-onedrive so maybe implementing HTTP requests directly from API docs may be easier.
Onedrive API is very straightforward so it won't be a problem to implement limited subset of the API calls:

Add comments above exported functions

or don't

The package log is very limited. Only has Print, Fatal, Panic methods and I think the usage of Fatal and Panic should be avoided.

With fmt, we get to separate stdin and stderr outputs but still, it would be nice to have more indicators on logs.

So I suggest using one of the logging libraries here. It would be nice to have debug, info, warning and error levels.

debug is for verbose messages like the ones in the fs.go:

info should be used for information obviously :D. We don't have those in the code right now but I think we can use it for the messages like 'file x is uploaded to y'.

warning should be used for handled errors. For example, the error here is handled:

error is for fatal errors that shouldn't happen. There we can log as error and do a clean exit (unmount, wait for ongoing network operations, etc.)

CBC

• When done right it's very good
• No parallel encryption
• Needs padding

CTR

• Parallel encryption
• "Some question the security of the "related plaintext" model " [1]

1: https://security.stackexchange.com/questions/52665/which-is-the-best-cipher-mode-and-padding-mode-for-aes-encryption

Change default log level to info and add --debug parameter for debug logs. When cloudstash is run with --debug option:

• it should set log level of dropbox to debug
• it should mount FUSE filesystem with -d option
• it should set log level of logrus accordingly