falense / b2_fuse

FUSE integration for Backblaze B2 Cloud storage

b2_fuse - FUSE for Backblaze B2

Version: 1.3

This is fresh release based upon B2 Command line tool's API for B2.

Warning this software may contain bugs, be careful of using it with important data.

Please report bugs, use-case issues and feature requests through the Github issue tracker

Basic setup:

Requires YAML and FUSE for python to work (this is not the same as "python-fuse" package).

Python2 setup

Install YAML for python as follows:

sudo apt-get install python-yaml

Install via pip:

sudo pip install b2fuse

Or install from code in development mode by executing in the base directory of this repository:

sudo python setup.py develop

If the above does not work (a possibility on Python 2.7), you can also install FUSE and the B2 Command Line Tool dependencies as follows:

sudo python -m pip install fusepy b2

Python3 setup

Install dependencies for Python3 as follows:

sudo python -m pip install fusepy b2 pyyaml

Install B2Fuse itself for Python3:

sudo python setup.py install

Configuration and usage

You will need a config.yaml file in the folder where you run the fuse driver. An example config ("config.yaml"):

accountId: <youraccountid>
applicationKey: <yourapplicationid>
bucketId: <yourbucketid>

In order to use the FUSE driver as an interface to the online service B2 run:

b2fuse <mountpoint>

Full usage info:

usage: b2fuse [-h] [--enable_hashfiles] [--version] [--use_disk] [--debug]
              [--account_id ACCOUNT_ID] [--application_key APPLICATION_KEY]
              [--bucket_id BUCKET_ID] [--temp_folder TEMP_FOLDER]
              [--config_filename CONFIG_FILENAME] [--allow_other]
              mountpoint

positional arguments:
  mountpoint            Mountpoint for the B2 bucket

optional arguments:
  -h, --help            show this help message and exit
  --enable_hashfiles    Enable normally hidden hashes as exposed by B2 API
  --version             show program's version number and exit
  --use_disk
  --account_id ACCOUNT_ID
                        Account ID for your B2 account (overrides config)
  --application_key APPLICATION_KEY
                        Application key for your account (overrides config)
  --bucket_id BUCKET_ID
                        Bucket ID for the bucket to mount (overrides config)
  --temp_folder TEMP_FOLDER
                        Temporary file folder
  --config_filename CONFIG_FILENAME
                        Config file

Usage notes:

• Can be used as a regular filesystem, but should not (high latency)
• Files are cached in memory or on disk. If using memory you are limited by the available memory, swapping will occur for very large files.
• Neither permissions or timestamps are supported by B2. B2_fuse ignores any requests to set permissions.
• Filesystem contains ".sha1" files, these are undeletable and contain the hash of the file without the postfix. This feature can be disabled by setting variable "enable_hashfiles" to False.
• For optimal performance and throughput, you should store a few large files. Small files suffer from latency issues due to the way B2 API is implemented. Large files will allow you to saturate your internet connection.

Testing

Go to the repostiory root directory, create a folder called mountpoint and run the test by calling

python -m "b2fuse.tier1_tests"

Application specific notes:

Using RSync with B2 Fuse

Since there is no support for updating file times or permissions in a bucket, rsync must be told to ignore both when synching folders (sync will be based on checksum meaning files have to be downloaded to compare).

rsync -avzh --no-perms --no-owner --no-group dir1/ dir2/ 

Option "--inplace" may also be useful. RSync creates a temporary file when syncing, this option will make RSync update the file inplace instead (Effectively twice as fast syncing).

Using unison to synchronize against mounted folder

Again, we ignore permissions as these are not applicable.

unison dir1/ dir2/ -auto  -perms 0  -batch

Using encfs to overlay a locally encrypted filesystem onto the bucket

Install encfs (apt-get install encfs)

encfs <bucket_mountpoint> <encrypted_filesystem>

Instruction for permanent mounting of B2Fuse can be found here

Known issues:

• Concurrent access from multiple client will lead to inconsistent results
• Small files give low read/write performance (due to high latency)
• Some text editors may return odd characters when viewing the ".sha1" files. Use cat for reliable operation.

License: MIT license

b2_fuse's Issues

Auto Mount on boot

Is there a way to mount this automatically on boot and also keep it in the backgroud without displaying it's continuous output?

Cannot view contents of bucket

I want to store some backups on Backblaze as a disaster recovery option.
I am running the python script in a screen so that it stays mounted.
I can create directories in there - works great.

However, when I unmount and remount the b2 storage it appears to be empty. Is this the expected behavior?

Local 'mv' results in empty target file

Saving a file to the b2_fuse mount point successfully pushes the file to BackBlaze B2 storage.

Issuing a local 'mv' command (Eg. mv ubuntu.iso ubuntu.iso.old) results in the target file being created with size 0KB and the source file being deleted.

Here is the DEBUG log from the 'mv' command:

2016-02-16 19:40:23,375:INFO:_list_dir 2016-02-16 19:40:23,376:DEBUG:Rename old: /ubuntu-14.04.3-desktop-amd64.iso?_ga=1.259617613.699603941.1455208484, new /ubuntu.iso 2016-02-16 19:40:23,376:INFO:_list_dir 2016-02-16 19:40:23,376:INFO:_list_dir 2016-02-16 19:40:23,376:DEBUG:Flush ubuntu.iso 0 2016-02-16 19:40:23,376:INFO:put_file ubuntu.iso (len:0) 2016-02-16 19:40:23,376:INFO:_list_dir 2016-02-16 19:40:25,176:INFO:put_file Upload complete 2016-02-16 19:40:25,176:DEBUG:Unlink ubuntu-14.04.3-desktop-amd64.iso?_ga=1.259617613.699603941.1455208484 2016-02-16 19:40:25,176:INFO:_list_dir 2016-02-16 19:40:25,176:INFO:delete_file ubuntu-14.04.3-desktop-amd64.iso?_ga=1.259617613.699603941.1455208484 2016-02-16 19:40:25,177:INFO:get_file_versions ubuntu-14.04.3-desktop-amd64.iso?_ga=1.259617613.699603941.1455208484 2016-02-16 19:40:40,917:DEBUG:Readdir /

Only first 100 files in alphabetical order can be seen

Issue description

When listing files in a mount, only the first 100 files in alphabetical order are listed. Also, any attempts to upload files with a filename after the first 100 files alphabetically will fail.

Steps to reproduce the issue

1. Create and mount an empty bucket
2. Enter the mount and generate 128 files with the following:
   touch `seq -w 1 128`
3. Unmount and re-mount the bucket
4. List the files in the bucket with ls

What's the expected result?

• 128 files from 001 to 128 should be in the directory listing

What's the actual result?

• Only files 001 to 100 are in the directory listing. Files 101 to 128 are missing.

Remove memory limit command line argument

This argument doesn't do anything. TODO

Error parsing config file, related to python version (2.7 vs 3)?

python b2fuse.py /home/nodejs/bucket
Traceback (most recent call last):
File "b2fuse.py", line 499, in
main(args.mountpoint, args.account_id, args.application_key, args.bucket_id)
File "b2fuse.py", line 491, in main
with B2Fuse(account_id, application_key, bucket_id) as filesystem:
File "b2fuse.py", line 93, in init
config = load_config()
File "b2fuse.py", line 86, in load_config
return yaml.load(f.read())
File "/usr/lib/python2.7/dist-packages/yaml/init.py", line 71, in load
return loader.get_single_data()
File "/usr/lib/python2.7/dist-packages/yaml/constructor.py", line 37, in get_single_data
node = self.get_single_node()
File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 35, in get_single_node
if not self.check_event(StreamEndEvent):
File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 98, in check_event
self.current_event = self.state()
File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 143, in parse_implicit_document_start
StreamEndToken):
File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 116, in check_token
self.fetch_more_tokens()
File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 257, in fetch_more_tokens
% ch.encode('utf-8'), self.get_mark())
yaml.scanner.ScannerError: while scanning for the next token
found character '\t' that cannot start any token
in "", line 1, column 23:
accountId:SecretXXXXXXX
^

Getting URLError

My log keeps getting filled with URLErrors when using b2local.py. It downloads files fine, but it doesn't seem to upload any files.

Exception in thread Thread-810:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/b2_fuse/b2local.py", line 211, in _download_thread
    self.download_file(b2_filename, local_filename)
  File "/opt/b2_fuse/b2local.py", line 185, in download_file
    data = self.bucket.get_file(b2_filename)
  File "/opt/b2_fuse/b2bucket_threaded.py", line 134, in get_file
    return super(B2BucketThreaded,self).get_file(filename)
  File "/opt/b2_fuse/b2bucket.py", line 214, in get_file
    return self._get_file(*args)
  File "/opt/b2_fuse/b2bucket.py", line 181, in _get_file
    with OpenUrl(api_url, None, encoded_headers) as resp:
  File "/opt/b2_fuse/b2_python_pusher.py", line 61, in __enter__
    self.file = urllib2.urlopen(request)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno -2] Name or service not known>

503 errors not handled

Hi,

According to Backblaze it is common to receive a 503 response to a request if a storage pod is busy.
From this page on uploading:

// Try several times to upload the file.  It's normal
// for uploads to fail if the target storage pod is
// too busy.  It's also normal (but infrequent) to get
// a 429 Too Many Requests if you are uploading a LOT
// of files.

Here is the output from b2_fuse on such a response:

Headers: {'Content-Type': 'b2/x-auto', 'X-Bz-File-Name': 'data/0/839', 'X-Bz-Content-Sha1': '02894f9352d498d39517c01c2557cd8555a0bb4d', 'Content-Length': '13
99322', 'Authorization': '***********************************************************'}

{
  "code": "service_unavailable",
  "message": "c001_v0001001_t0037 is too busy",
  "status": 503
}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
    return func(*args, **kwargs) or 0
  File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 621, in flush
    return self.operations('flush', self._decode_optional_path(path), fh)
  File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in __call__
    return getattr(self, op)(*args)
  File "b2fuse.py", line 456, in flush
    self.bucket.put_file(path, self.open_files[path])
  File "/home/mystx/b2_fuse/b2bucket_cached.py", line 149, in put_file
  File "/home/mystx/b2_fuse/b2bucket.py", line 148, in _put_file
  File "/home/mystx/b2_fuse/b2_python_pusher.py", line 71, in __enter__
SystemExit: 1

This causes my backup program to stop. Is there a way to get b2_fuse to retry?

Thanks

Error: fuse: device not found, try 'modprobe fuse' first

I have an error when trying to mount:

# python b2fuse.py /mnt/backblaze
2016-07-22 22:22:07,903:INFO:list_buckets
fuse: device not found, try 'modprobe fuse' first
Traceback (most recent call last):
  File "b2fuse.py", line 499, in <module>
    main(args.mountpoint, args.account_id, args.application_key, args.bucket_id)
  File "b2fuse.py", line 492, in main
    FUSE(filesystem, mountpoint, nothreads=True, foreground=True)
  File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 480, in __init__
    raise RuntimeError(err)
RuntimeError: 1

config.yaml:

# cat config.yaml 
accountId: "id"
applicationKey: "key"
bucketId: "id"

Also, now that BB support files with any size. is b2 fuse only limited to the ram? If so have you had any experience in using a ram drive for this. What i would like to use BB for is to extend the VPS storage for a NAS.

Optimize large file uploads

By making use of large file upload features in B2 it should be possible to reduce the memory footprint of B2Fuse to a set maximum memory size. This at the cost of disk space as the entire file must be stored somewhere. Instead of loading the entire file to memory before uploading, chunks could be loaded at a time limiting the maximal memory consumption at any given time.

Hangs on "list_buckets"

This is an awesome looking tool, and I can't wait to use it. Thank you for creating it.

When attempting to mount B2, the software hangs at "list_buckets"

$ sudo python b2fuse.py /mnt/b2
2016-04-07 08:48:30,571:INFO:list_buckets

The bucket contains no files.

ImportError: cannot import name FUSE

I am trying to figure out a problem with the fuse library. I have installed the library in several ways, but still get an import error when trying to run the code. Can you provide any insight based on this screen shot? OS is Ubuntu 14.04 LTS.

Reading hash files results in "bad address"

Reading hash files results in "bad address"

1000 files per bucket limit

Thanks for b2_fuse !

I'd like to backup my music collection to B2, but this limitation makes it impossible as I have around 50,000 files in it. Why is the limitation? As I understood here, the API allows me to upload unlimited amount of files per bucket.

Anything I can do to solve it?

Thanks again!

Using rsync to upload files causes my download to increase

I was trying to test backing up my music library to B2, so I mounted my bucket and kicked off an Rsync of about 12 FLAC albums. My 1 GB of free downloads got used up in the process.

If I am using rsync to sync files up to b2, how did it used a 1 GB of download bandwidth?

How to run b2_fuse in a Docker container?

I'm using AWS S3 at this point and I would like to use another Datacenter as well. As I'm a Docker fan, I'm wondering if anyone created a container out of this project?

If this can inspire anyone else, here is my Dockerfile for AWS CLI (S3):

FROM phusion/baseimage:0.9.19

# Install these apps
RUN \
    apt-get update && apt-get install -y \
    nano \
    rsync \
    unzip \
    gzip \
    wget \
    openssl \
    ntp \
    secure-delete \
    git \
    git-core \
    python2.7 \
    curl \
    groff && \
    curl -O https://bootstrap.pypa.io/get-pip.py && \
    python3 get-pip.py && \
    pip install awscli && \
    aws help && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    echo "apt-get install is finished" && \
    echo

WORKDIR /root

# Use baseimage-docker's init system.
CMD ["/sbin/my_init"]

Option “--enable_hashfiles” and .sha1 files

In README.md says that “.sha1” files can be disabled by setting variable "enable_hashfiles" to False but:

Setting “--enable_hashfiles false” or “--enable_hashfiles=false” is not accepted. The same for “true”.

It looks that enabling “.sha1” files is already disabled by default and the option “--enable_hashfiles” enables them.

So, if I am right, README.md should be modified accordingly.

Most important: after enabling “.sha1” files with “--enable_hashfiles” if I try to read it with “cat” or copy it to the hard disk the console displays the message “Bad address”. In the case of the copy command it copies a hash file which size is 0, despite the size in the mounted filesystem its size appears to be 42 bytes. If I try to edit a hash file in the mounted filesystem with nano, the console displays the message “Too many open files in system”. In all cases b2fuse.py displays these errors:

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 591, in read
data = create_string_buffer(ret, retsize)
File "/usr/lib/python2.7/ctypes/init.py", line 68, in create_string_buffer
raise TypeError(init)
TypeError: array('c', '85beca341f971884732121a9083b1253f9ce33e4\n')
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 591, in read
data = create_string_buffer(ret, retsize)
File "/usr/lib/python2.7/ctypes/init.py", line 68, in create_string_buffer
raise TypeError(init)
TypeError: array('c', '85beca341f971884732121a9083b1253f9ce33e4\n')

If Backblaze question and the article it references to are right this is an important matter:

https://help.backblaze.com/hc/en-us/articles/218020298-Why-does-B2-require-a-SHA-1-hash-to-be-provided-with-an-upload-

https://stackoverflow.com/questions/3830206/can-a-tcp-checksum-produce-a-false-positive-if-yes-how-is-this-dealt-with

Please, add a parameter to display the branch and release of the program

If we have different branches and releases of the program installed it would be very good that b2fuse.py reported what its branch and release is with something like "b2fuse --version" or similar. Right now, the only way I can devise is to check the name of the folder b2fuse is placed in, which is not optimal IMO.

I have almost finished a bash script that will conveniently allow the use of different branches and releases that could do a good use of this. This would allow people to use the latest stable release to safely store their important files in a bucket and use another bucket to test the latest released master branch or other alternative branches.

Cannot remove directory with filename that is a prefix of an existing filename

Issue description

An error occurs when I try to rmdir a directory that has a filename which is a prefix of the filename of an existing file.

Steps to reproduce the issue

1. Create file "abcd"
2. Create directories "a", "ab", and "abc"
3. Remove the directories created in step 2

What's the expected result?

• Directories are successfully removed

What's the actual result?

• Directories fail to be removed with the following traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuse.py", line 495, in _wrapper
    return func(*args, **kwargs) or 0
  File "/usr/lib/python2.7/site-packages/fuse.py", line 533, in rmdir
    return self.operations('rmdir', path.decode(self.encoding))
  File "/usr/lib/python2.7/site-packages/fuse.py", line 800, in __call__
    return getattr(self, op)(*args)
  File "/opt/b2_fuse/b2fuse_main.py", line 320, in rmdir
    self.api.delete_file_version(fileName_to_fileId[path], path)
KeyError: u'abc'

Where "abc" is the name of the directory to be removed.

question on date on files

I am enjoying using your fuse for b2, but I am having a rather odd issue. I can mount the filesystem and read and write files. The folder show the correct timestamp (as it shows directly in the b2 web interface, but the files (non-directories) all have odd dates - various dates sometimes around 1969 other ones show up as July 18 1982.

Have you seen this before or have any ideas what would cause it?

I am running ubuntu 16.04 LTS.

Thanks

SHA1 files appear on directory listing

Sondre- moving our conversation to github.:)

When I mount a bucket and ls the directory - I'm seeing the a file with the extension .sha1 for each file in my directory. When I try to cat the file, I get the following error: cat: img.jpg.sha1: Bad address.

-- Nilay

Adding support for "mtime" using extended file metadata

rclone has solved the mtime problem by storing them as metada:

The modified time is stored as metadata on the object as X-Bz-Info-src_last_modified_millis as milliseconds since 1970-01-01 in the Backblaze standard. Other tools should be able to use this as a modified time.

If b2_fuse were to do exactly the same, it wouldn't just make the mount a lot better, but also be compatible with rclone.

Thank you very much for considering.

expired_auth_token error after running it for couple days

I mounted for couple days then tried to copy a file to it, and got the following error

I used python 2.7.11 and

command

"b2fuse.py /mnt/mountpoint"

`2016-05-12 23:37:53,498:INFO:_list_dir
Error returned from server:

URL: https://api001.backblaze.com/b2api/v1/b2_list_file_names
Params: {"bucketId": "4f709ffffb2796275f4d0219", "startFileName": "", "maxFileCount": 1000}
Headers: {'Authorization': u'3_20160510072929_5c2daf8dd724a7e42cd1009b_b88d1324234243813534dfa81df3aaab10a0b74d3a9d99a24_001_acct'}

{
"code": "expired_auth_token",
"message": "Authorization token has expired",
"status": 401
}
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(_args, *_kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 511, in getattr
return self.fgetattr(path, buf, None)
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 759, in fgetattr
attrs = self.operations('getattr', self._decode_optional_path(path), fh)
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "b2fuse.py", line 174, in getattr
elif self._exists(path):
File "b2fuse.py", line 131, in _exists
if path in self.bucket.list_dir():
File "/mnt/redline/Projects/python/b2_fuse/b2_fuse/b2bucket_cached.py", line 118, in list_dir
return map(lambda x: x['fileName'], self._list_dir(startFilename))
File "/mnt/redline/Projects/python/b2_fuse/b2_fuse/b2bucket_cached.py", line 114, in _list_dir
result = super(B2BucketCached, self)._list_dir(startFilename)
File "/mnt/redline/Projects/python/b2_fuse/b2_fuse/b2bucket.py", line 84, in _list_dir
resp = call_api(self.api_url, api_call, self.account_token, call_params)
File "/mnt/redline/Projects/python/b2_fuse/b2_fuse/b2_python_pusher.py", line 85, in call_api
with OpenUrl(url, request_data, headers) as f:
File "/mnt/redline/Projects/python/b2_fuse/b2_fuse/b2_python_pusher.py", line 71, in enter
sys.exit(1)
SystemExit: 1
`

Purge or Refresh the Cache for a File

This seems to work very well. But I would like to deploy this in a way that two clients are accessing files on the same bucket. If client A changes to or deletes a file, client B should know about the change and refresh it's cache.

I can handle notifying each client connected to the bucket. The problem is what would be the best way to force refreshing the cache of a particular file? Deleting a files from the --temp_folder causes the file system to break and I have to restart it. No errors are thrown when this happens. I imagine when it sees the cache is missing, it raises an error, and doesn't successfully recover.

It would be nice if we had a way to enable Write Cache support like s3ql does. It would be a successful write when it finishes writing to cache, and let the file system finish syncing the data in the background. It may also be useful to be able to setup some sort of callback after the cached file has finished it's upload to the bucket. With the way Backblaze handles versioning, conflicts shouldn't be an issue if multiple clients happen to write to a given file at the same time.

I am possibly considering extending the file system to have some sort of api to access functions like clearing the cache for a file. It could make for a great framework to build a replicated geo-caching filesystem, with low-cost archive storage on the backend. Not sure if an API would be beyond the scope of this project / should be a fork?

Adding tests

Hello!

You may steal these tests.
https://github.com/s3ql/s3ql/tree/master/tests

Add support for installation using PIP

TODO

Cannot create file or directory with filename that is a suffix of the filename of an existing file

Issue description

An error occurs if I create a file or directory with a filename that is a suffix of an existing file

Steps to reproduce the issue

1. Create a file named "abcd"
2. Create files or directories named "bcd", "cd", and "d"

What's the expected result?

• Files "abcd", "bcd", "cd", and "d" are created

What's the actual result?

• File "abcd" is created, but creating "bcd", "cd", and "d" fails with the following traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuse.py", line 495, in _wrapper
    return func(*args, **kwargs) or 0
  File "/usr/lib/python2.7/site-packages/fuse.py", line 511, in getattr
    return self.fgetattr(path, buf, None)
  File "/usr/lib/python2.7/site-packages/fuse.py", line 759, in fgetattr
    attrs = self.operations('getattr', self._decode_optional_path(path), fh)
  File "/usr/lib/python2.7/site-packages/fuse.py", line 800, in __call__
    return getattr(self, op)(*args)
  File "/opt/b2_fuse/b2fuse_main.py", line 220, in getattr
    st_size=len(self.open_files[path])
TypeError: __init__() takes at least 3 arguments (1 given)

Segmentation Fault - Out of memory

I just tried this for the first time and tried to do an ls on the mount point. I had python running in one terminal and my ls in the other. I could see the program getting the files from b2 in the python terminal. After a few minutes I got a segmentation fault.

Here's the bucket details:
Current Files: 435,185 Current GB: 635.111 GB

This is the only thing I see in the syslog that might be helpful (let me know if you want something else):
Mar 2 20:42:16 files kernel: [20445.742796] show_signal_msg: 30 callbacks suppressed Mar 2 20:42:16 files kernel: [20445.742801] python[13715]: segfault at 24 ip 0000000000558077 sp 00007fffd3521310 error 6 in python2.7[400000+2bc000] Mar 2 20:42:16 files kernel: [20445.763319] Core dump to |/usr/share/apport/apport 13715 11 0 13715 pipe failed

Could it be mounted as other FUSE

Could it be mounted as other FUSE with a line in fstab and the normal mount / fusermount -u commands?
(Sorry, I forgot the question mark in the title)

Trouble finding fuse / invalid token

I have fuse and libfuse installed via package repo, and I'm getting the following error:

# python b2fuse.py /srv/b2
Traceback (most recent call last):
  File "b2fuse.py", line 35, in <module>
    from fuse import FUSE, FuseOSError, Operations
ImportError: No module named fuse

I know im not supposed to use python-fuse I think, but installing it results in a slightly different error:

# python b2fuse.py /srv/b2
Traceback (most recent call last):
  File "b2fuse.py", line 35, in <module>
    from fuse import FUSE, FuseOSError, Operations
ImportError: cannot import name FUSE

The python3 command gives me this error:

# python3 b2fuse.py /srv/b2
  File "b2fuse.py", line 172
    return dict(st_mode=(S_IFDIR | 0777), st_ctime=time(), st_mtime=time(), st_atime=time(), st_nlink=2)
                                      ^
SyntaxError: invalid token

I'm pretty sure I'm just an idiot... any ideas? Using openSuSE 13.1.

Directories not showing in mount point

I can see the directories and files in the web interface of backblaze but, are unable to see all of them in the mounted folder.

Any idea?

mount folder not accessible after starting b2fuse.py for "Online only copy"

It's not feasible for me to utilize b2_fuse in the "With local copy" mode, (which works fine).
Therefore I have been attempting to utilize b2fuse.py as described in the documentation.
It runs (and gives me the status, repeated over time).
Unfortunately nothing can read from or write to the folder.
In fact the folder shows exactly the symptoms described in this Stack Exchange question: http://superuser.com/questions/169977/mount-point-permission-denied
Unfortunately what that user did to fix the problem does not directly translate.
I have set fuse.conf to allow_root and user_allow_other with no change (including after restarting the server and also waiting for an extended amount of time... hours...).

I have never utilized fuse before this so it could still be some configuration error on my part... if it is I apologize.

I started looking into the code and attempting to figure out if there was something that needed to be passed to libfuse that was missed (by looking @ the fuse api, then looking through the libfuse code then the b2fuse code)... I am not very proficient with python & decided to go a different direction (instead of attempting to get b2 mounted locally).

MORE DETAILS:
OS=Ubuntu 14.04.5 server (64bit)

Finish implement disk filetype for lower memory footprint

TODO

Alternative temp folder deleted after unmounting!!!

Setting the option "--temp_folder" to an alternative temp folder, after unmounting the filesystem, that temp folder is deleted and all its contents are lost. I have lost a folder with a bunch of good stuff in that way; fuck! fuck! FUCK!!!!!

It happens regardless it is unmounted with "fusermount -u" or with "sudo umount".

Please, correct this ASAP.

Problem to install on Python 2.7

When I try to install with “sudo python -m pip install fusepy“ I get this output:
/usr/bin/python: cannot import name IncompleteRead; 'pip' is a package and cannot be directly executed

Issues mounting bucket (first run)

python3 b2fuse.py /home/dzaczek/blaze
File "b2fuse.py", line 172
return dict(st_mode=(S_IFDIR | 0777), st_ctime=time(), st_mtime=time(), st_atime=time(), st_nlink=2)
^

'b2fuse.py': [Errno 2] No such file or directory

I installed everything via pip - but I cannot find the .py file to run in order to mount the bucket. Where is it?

I searching my entire drive using:

find / -name "b2fuse" -print0

...nothing came up!

Add support for streaming downloads

TODO

I must be doing something wrong...

I'm getting the following error when using python b2fuse.py /home/plex/b2

File "b2fuse.py", line 35, in <module> from fuse import FUSE, FuseOSError, Operations ImportError: No module named fuse
I have fusepy installed in /usr/local/lib/python3.5/dist-packages

Did I miss something completely?

Thank you for your help!

Errors when copying and deleting a folder

I've got some errors when copying and deleting a folder with “cp -r” and “rm -r” commands.

The testing folder is 4.5 MB big with 94 files and 6 sub-folders. This is the output of the “tree” program on it:

JRRTolkien.LoshijosdeHurin
├── META-INF
│   └── container.xml
├── mimetype
└── OEBPS
├── content.opf
├── Fonts
│   └── tolkien.ttf
├── Images
│   ├── autor.jpg
│   ├── cap01img.jpg
│   ├── cap01.jpg
│   ├── cap02.jpg
│   ├── cap03.jpg
│   ├── cap04.jpg
│   ├── cap05img.jpg
│   ├── cap05.jpg
│   ├── cap06img.jpg
│   ├── cap06.jpg
│   ├── cap07.jpg
│   ├── cap08.jpg
│   ├── cap09.jpg
│   ├── cap10.jpg
│   ├── cap11img.jpg
│   ├── cap11.jpg
│   ├── cap12img.jpg
│   ├── cap12.jpg
│   ├── cap13img.jpg
│   ├── cap13.jpg
│   ├── cap14img.jpg
│   ├── cap14.jpg
│   ├── cap15.jpg
│   ├── cap16.jpg
│   ├── cap17.jpg
│   ├── cap18img.jpg
│   ├── cap18.jpg
│   ├── cover.jpg
│   ├── epubgratis.png
│   ├── ePUBlogo.png
│   ├── img01.jpg
│   ├── img02.jpg
│   ├── img03.jpg
│   ├── img04.jpg
│   ├── img05.jpg
│   ├── img06.jpg
│   ├── img07.jpg
│   ├── img08.jpg
│   ├── img09.jpg
│   ├── img10.jpg
│   ├── img11.jpg
│   ├── narnimg.jpg
│   └── notasimg.jpg
├── Styles
│   └── style.css
├── Text
│   ├── apendice.xhtml
│   ├── autor.xhtml
│   ├── cap01.xhtml
│   ├── cap02.xhtml
│   ├── cap03.xhtml
│   ├── cap04.xhtml
│   ├── cap05.xhtml
│   ├── cap06.xhtml
│   ├── cap07a.xhtml
│   ├── cap07b.xhtml
│   ├── cap08.xhtml
│   ├── cap09.xhtml
│   ├── cap10a.xhtml
│   ├── cap10b.xhtml
│   ├── cap11.xhtml
│   ├── cap12.xhtml
│   ├── cap13.xhtml
│   ├── cap14.xhtml
│   ├── cap15.xhtml
│   ├── cap16a.xhtml
│   ├── cap16b.xhtml
│   ├── cap17.xhtml
│   ├── cap18a.xhtml
│   ├── cap18b.xhtml
│   ├── cubierta.xhtml
│   ├── img01.xhtml
│   ├── img02.xhtml
│   ├── img03.xhtml
│   ├── img04.xhtml
│   ├── img05.xhtml
│   ├── img06.xhtml
│   ├── img07.xhtml
│   ├── img08.xhtml
│   ├── img09.xhtml
│   ├── img10.xhtml
│   ├── img11.xhtml
│   ├── info.xhtml
│   ├── introduccion1.xhtml
│   ├── introduccion2.xhtml
│   ├── narn.xhtml
│   ├── notas.xhtml
│   ├── prefacio.xhtml
│   ├── pronunciacion.xhtml
│   ├── sinopsis.xhtml
│   └── titulo.xhtml
└── toc.ncx

Problem 1: entries in the bucket that disppear.

The top level of the test bucket has 24 entries (files and folders) before copying the test folder. After the copy of the test folder with “cp -r” some files and folders disappear, leaving one folder and seven files plus the newly copied test folder and no error output by b2fuse.py. So, in the report of the second problem below, when I write “cp -r no error” I mean that b2fuse.py doesn't output any error, but this problem always happens. But the files and folders are there; they can be seen using the web interface and they appear again after deleting the testing folder.

I wonder if this would be about the maximum number of files that b2fuse can handle. The bucket has 103 files and 10 subfolders and the test folder I copy has 94 files and 6 subfolders. The sum is 197 files and 16 subfolders. Are these too many entries for b2fuse?

Problem 2: errors when deleting a folder with “rm -r”

I have repeated several times the copy and delete of the testing folder with “cp -r” and “rm -r”. These are the errors reported by b2fuse.py.

cp -r no error
rm -r no error

cp -r no error
rm -r

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 693, in readdir
fip.contents.fh):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 251, in readdir
online_files = map(lambda file_info: file_info['fileName'], directory.get_file_infos())
AttributeError: 'NoneType' object has no attribute 'get_file_infos'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 693, in readdir
fip.contents.fh):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 251, in readdir
online_files = map(lambda file_info: file_info['fileName'], directory.get_file_infos())
AttributeError: 'NoneType' object has no attribute 'get_file_infos'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 693, in readdir
fip.contents.fh):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 251, in readdir
online_files = map(lambda file_info: file_info['fileName'], directory.get_file_infos())
AttributeError: 'NoneType' object has no attribute 'get_file_infos'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 693, in readdir
fip.contents.fh):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 251, in readdir
online_files = map(lambda file_info: file_info['fileName'], directory.get_file_infos())
AttributeError: 'NoneType' object has no attribute 'get_file_infos'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 693, in readdir
fip.contents.fh):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 251, in readdir
online_files = map(lambda file_info: file_info['fileName'], directory.get_file_infos())
AttributeError: 'NoneType' object has no attribute 'get_file_infos'

cp -r no error
rm -r no error

cp -r no error
rm -r

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 693, in readdir
fip.contents.fh):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 251, in readdir
online_files = map(lambda file_info: file_info['fileName'], directory.get_file_infos())
AttributeError: 'NoneType' object has no attribute 'get_file_infos'

cp -r no error
rm -r no error

cp -r no error
rm -r

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 693, in readdir
fip.contents.fh):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 251, in readdir
online_files = map(lambda file_info: file_info['fileName'], directory.get_file_infos())
AttributeError: 'NoneType' object has no attribute 'get_file_infos'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 693, in readdir
fip.contents.fh):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 251, in readdir
online_files = map(lambda file_info: file_info['fileName'], directory.get_file_infos())
AttributeError: 'NoneType' object has no attribute 'get_file_infos'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 693, in readdir
fip.contents.fh):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 251, in readdir
online_files = map(lambda file_info: file_info['fileName'], directory.get_file_infos())
AttributeError: 'NoneType' object has no attribute 'get_file_infos'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 693, in readdir
fip.contents.fh):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 251, in readdir
online_files = map(lambda file_info: file_info['fileName'], directory.get_file_infos())
AttributeError: 'NoneType' object has no attribute 'get_file_infos'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 693, in readdir
fip.contents.fh):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 251, in readdir
online_files = map(lambda file_info: file_info['fileName'], directory.get_file_infos())
AttributeError: 'NoneType' object has no attribute 'get_file_infos'

Now, reading this, I wonder if there is a pattern:
cp/rm no-error
cp/rm error
cp/rm no-error
cp/rm error

But to be sure such pattern is real and not by chance I woud have to repeat it at least 3 or 4 times more.

Random access write not implemented, Borg backup fails with NotImplementedType

I wonder if this issue would be related to issue #38.

I' ve tried to do a test backup using Borg backup stable version (1.0.8): https://borgbackup.readthedocs.io/en/stable/index.html

I've got these errors:

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 604, in write
offset, fh)
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 431, in write
self.open_files[path].write(offset, data)
File "/home/durr/b2/b2_fuse/filetypes/B2SequentialFileMemory.py", line 72, in write
raise NotImplemented()
TypeError: 'NotImplementedType' object is not callable

The backup failed.

Unhandled error caused by malformed file access sequence

I've got these errors in the master branch, but unfortunately I don't know what I was doing when the errors appeared. Therefore, I can not reproduce them. I will report as soon as I get similar errors and know what I was doing to triggering them:

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 604, in write
offset, fh)
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 430, in write
self.open_files[path].set_dirty(True)
TypeError: init() takes at least 3 arguments (1 given)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 604, in write
offset, fh)
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 430, in write
self.open_files[path].set_dirty(True)
TypeError: init() takes at least 3 arguments (1 given)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 604, in write
offset, fh)
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 430, in write
self.open_files[path].set_dirty(True)
TypeError: init() takes at least 3 arguments (1 given)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 621, in flush
return self.operations('flush', self._decode_optional_path(path), fh)
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 445, in flush
self.open_files[self._remove_start_slash(path)].upload()
TypeError: init() takes at least 3 arguments (1 given)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 629, in release
return self.operations('release', self._decode_optional_path(path), fh)
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in call
return getattr(self, op)(*args)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 451, in release
self.flush(self._remove_start_slash(path), fh)
File "/home/durr/b2/b2_fuse/b2fuse_main.py", line 445, in flush
self.open_files[self._remove_start_slash(path)].upload()
TypeError: init() takes at least 3 arguments (1 given)

Encoding issue for hashfiles using “--enable_hashfiles” option

As the other issue with “--enable_hashfiles” is closed I've opened this one, but I'm not sure if I should have posted this in that issue despite being a closed one. Plese, tell me if was wrong to open a new issue on this.

Great work solving the former issue with this. But I still see minor problems in my system:

When openening the .sha1 files with nano sometimes I get a single line with its hash and sometimes I get also a second line with the characters “^@” (without “”). It looks like if it were random: sometimes I get the single line and sometimes the aforementioned two lines. Trying with okular program sometimes I get a second line with the word “nul” inside a sqare. With the program “geany” (a text editor intended for programming) I consistently get a line of chinese or japanese characters; this is one example: 㔸敢慣㐳昱㜹㠱㐸㌷ㄲㄲ㥡㠰戳㈱㌵㥦散㌳㑥.

Opening the .sha1 files with kdiff3 it always complains that it can not open the file. No problem with diff, cmp nor wdiff.

In all of these cases there is no error message from b2fuse.py.

Files not uploading to bucket

I have installed and configured b2fuse and it appears to be configured properly but files in the mount point aren't uploading to the bucket. the OS is CentOS7. It might be permissions but I have set them as per the instructions. Any help would be greatly appreciated.

Partition used space reads 0 bytes used

Hello!
Awesome project here that I like, able to mount as a filesystem with ease.

However, a quick look at df -h in my server shows an issue of the used disk space reads a constant 0 bytes, while over on my buckets page shows a true value.

Filesystem Size Used Avail Use% Mounted on
B2Fuse---- 64P 0---- 64P--- 0% /var/nfs/backblaze-s3

Issues with Bacula and B2Fuse from lack of permission support

When i running bacula to save backup volume files at mount point of b2fuse filesystem is displayed volume 0 byte and jobs still running for long time with not able to open files are created with no such file or directory , also not able to chown any sub folder at mybucket with error read-only file system although my mount point as your see below
B2Fuse on /mnt/bb type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

cannot get b2-fuse mount

When running the command:

myserver b2_fuse-master # python b2fuse.py --config_filename config.yaml --memory_limit 4G /mnt/b2

I get the following output.

Traceback (most recent call last):
File "b2fuse.py", line 32, in
from b2fuse_main import B2Fuse
File "/root/b2-fuse/b2_fuse-master/b2fuse_main.py", line 178
st_mode=(S_IFDIR | 0777),
^
SyntaxError: invalid token

OS - Gentoo
Kernel - 4.4.43
PIP & Python modules are all installed.

Permission Issues

How can I pass allow_other or otherwise allow other users to read and write to a mounted folder?

Mounting squashfs image via b2_fuse causes entire file download

Hello @Sondree!

Thanks for your excellent work.

I'm trying to mount a squashfs image from b2 bucket mounted with b2_fuse, but this makes b2_fuse to download entire image. I thought that would not be necessary since implementation of 'range' parameter to avoid this.

I mount image that is 186 MiB long, but I intend to move my entire archive of half of my life (just under 1 TiB in size), mount over b2_fuse, and... (put anything here).

This would solve many issues with lack of attributes / timestamp support, only 1000 items returning in file list etc. This might in turn boost sales of B2 product to variety of new customers / integrators I guess.

But there is no point in mounting 1 TiB image if it requires prior download to one's desktop / laptop / mobile phone, right?

Thanks again. Stay safe.
Marcin.

Converting b2_fuse into an S3QL backend

Please read the comments on Backblaze/B2_Command_Line_Tool#273

S3QL has a lot of features like tests, local caching, deduplication, encryption etc.
https://bitbucket.org/nikratio/s3ql/

Please consider adding B2 backend to S3QL
https://bitbucket.org/nikratio/s3ql/issues/181/add-backend-backblaze